# DM_LaneGCN
**Repository Path**: junhuisirup/lanegcn_deployment
## Basic Information
- **Project Name**: DM_LaneGCN
- **Description**: lanegcn模型部署
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-06-26
- **Last Updated**: 2026-07-03
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# LaneGCN: ONNX / TensorRT Deployment Pipeline
> A production-oriented deployment showcase for [LaneGCN](https://github.com/uber-research/LaneGCN) (ECCV 2020 Oral, Uber ATG).
[Paper](https://arxiv.org/pdf/2007.13732) | [Slides](http://www.cs.toronto.edu/~byang/slides/LaneGCN.pdf) | [ECCV 2020 Oral Video](https://yun.sfo2.digitaloceanspaces.com/public/lanegcn/video.mp4)
> **Original authors:** Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, Raquel Urtasun
> **Rank 1st** — [Argoverse Motion Forecasting Competition](https://evalai.cloudcv.org/web/challenges/challenge-page/454/leaderboard/1279)
***
## Table of Contents
- [About This Repository](#about-this-repository)
- [Demo Inference Visualizations](#demo-inference-visualizations)
- [Benchmark Results](#benchmark-results)
- [Repository Structure](#repository-structure)
- [Deployment Pipeline](#deployment-pipeline)
- [Key Engineering Contribution: Eliminating Shape Input Tensors](#key-engineering-contribution-eliminating-shape-input-tensors)
- [Quick Start (Deployment Only)](#quick-start-deployment-only)
- [Key Engineering Notes](#key-engineering-notes)
- [License](#license)
- [Citation](#citation)
***
## About This Repository
This repository is a **deployment-focused fork** of LaneGCN. It wraps the original Uber LaneGCN as an unmodified git submodule and adds a complete **ONNX / TensorRT deployment toolchain** on top — refactoring the model for export, building TensorRT engines with dynamic shapes, and running low-latency inference.
> **For training, testing, and data preparation, please refer to the original** **[`LaneGCN/README.MD`](LaneGCN/README.MD).** This README focuses exclusively on the deployment pipeline added by this fork.
***
## Demo Inference Visualizations
The following six scenes are full-pipeline TensorRT inference results produced by [`demo_inference.py`](demo_inference.py) on the Argoverse forecasting validation set. Each animation shows, in the agent's local frame, the observed 2 s history (blue), the ground-truth 3 s future (green), the top-K predicted trajectories with confidence scores, and the lane-graph context used by the model. Engines used: `trt_engines_w8a32/lanegcn.engine` (weight-only INT8 / FP32 accum).
Scene 03189
 |
Scene 05412
 |
Scene 09481
 |
Scene 11605
 |
Scene 11803
 |
Scene 12775
 |
> **Color legend (per frame):** blue = observed trajectory (history), green = ground-truth future, red/yellow/… = predicted future candidates (color encodes the top-K rank; top-1 is the highest-confidence prediction), grey = lane-graph context.
These clips are the actual on-device outputs from the deployed engine — they validate that the ONNX graph, the TensorRT engine, and the input preprocessor agree numerically (also cross-checked by [`_verify_aligned.py`](_verify_aligned.py)).
***
## Benchmark Results
Latency, throughput, and engine size across **5 backends** on the same input sample (seq 110, 100 iterations after warmup). All backends share the same 23-tensor feed and use `time.perf_counter` for unified timing — see [`benchmark_all.py`](benchmark_all.py).
| # | Backend | Engine / Weights | Notes |
| --- | --- | --- | --- |
| 1 | **PyTorch eager** | `lanegcn_onnx.py` + `36.000.ckpt` | eval + `no_grad` + `cuda.synchronize` |
| 2 | **ONNX Runtime** | `lanegcn_simplified.onnx` | CUDAExecutionProvider |
| 3 | **TensorRT FP32** | `trt_engines_fp32/lanegcn.engine` | full-precision reference |
| 4 | **TensorRT w8a32** | `trt_engines_w8a32/lanegcn.engine` | INT8 weights + FP32 activations |
| 5 | **TensorRT w8a16** | `trt_engines_w8a16/lanegcn.engine` | INT8 weights + FP16 activations |
Reproduce with:
```sh
python benchmark_all.py --seq 110 --iterations 100 --output benchmark_results.json
```
***
## Repository Structure
The original [Uber LaneGCN](https://github.com/uber-research/LaneGCN) is included as a **git submodule** at `LaneGCN/`. The parent repository only contains the deployment-specific code and two compatibility-patched files:
```
LaneGCN_white/ ← this repo (deployment showcase)
├── LaneGCN/ ← git submodule (original Uber LaneGCN, unmodified)
│ ├── lanegcn.py ← original model
│ ├── utils.py ← original utils (imported via sys.path)
│ ├── train.py / test.py ← original training / testing scripts
│ ├── get_data.sh ← data download script
│ └── ...
├── lanegcn_onnx.py ← ★ NEW: ONNX-compatible model refactor
├── onnx_exporter.py ← ★ NEW: PyTorch → ONNX export
├── onnx_inference.py ← ★ NEW: ONNX Runtime benchmarking
├── trt_exporter.py ← ★ NEW: ONNX → TensorRT engine build
├── demo_inference.py ← ★ NEW: TRT inference + visualization
├── save_seq_data.py ← ★ NEW: debug data export
├── _verify_aligned.py ← ★ NEW: numerical alignment verification
├── data.py ← ★ MODIFIED: np.bool_ compat + _build_cross_edges + path config
├── layers.py ← ★ MODIFIED: math.gcd compat (Python 3.9+)
├── README.MD / LICENSE / .gitignore / requirements.txt / environment.yml
├── misc/ ← README images (for GitHub rendering)
└── fonts/ ← visualization font
```
**Why** **`data.py`** **and** **`layers.py`** **are duplicated:** These two files contain small but necessary patches on top of the original — `np.bool` → `np.bool_` (NumPy 1.24+), `fractions.gcd` → `math.gcd` (Python 3.9+), and a new `_build_cross_edges` method for online graph construction when `preprocess=False`. The deployment scripts set `sys.path` so the local (patched) versions take precedence over the submodule's originals.
***
## Deployment Pipeline
On top of the original LaneGCN, this repo adds a complete deployment toolchain:
| Stage | Script | Description |
| ------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| 1. Refactor | `lanegcn_onnx.py` | ONNX-compatible model: dict/list inputs flattened to 23 tensor inputs; graph preprocessing moved outside the model |
| 2. Export | `onnx_exporter.py` | PyTorch → ONNX with dynamic axes (variable actor/lane-node counts) |
| 3. Simplify | (onnx-simplifier) | Reduces ONNX graph for TRT compatibility |
| 4. Build | `trt_exporter.py` | ONNX → TensorRT engine (fp32 / fp16, dynamic shapes) |
| 5. Inference | `demo_inference.py` | TRT engine inference + visualization |
| 6. Profile | `onnx_inference.py` | ONNX Runtime latency / throughput benchmarking |
### Key Engineering Contribution: Eliminating Shape Input Tensors
The original LaneGCN uses dynamic slicing with `pre_mask` / `suc_mask` to select predecessor/successor lane-node pairs. These masks produce **shape input tensors** in the ONNX graph, which trigger a **segmentation fault in TensorRT 10.x** when called via `execute_async_v3`.
**Solution:** Replace dynamic slicing with valid-mask zeroing. In `lanegcn_onnx.py`:
```python
# Before (creates shape input tensors → TRT segfault):
edge_u = edge_u[pre_mask]
# After (mathematically equivalent, no shape inputs):
valid = pre_mask # boolean mask
edge_u_safe = torch.where(valid, edge_u, torch.zeros_like(edge_u))
```
This is mathematically equivalent and eliminates all shape input tensors, allowing `execute_async_v3` to run without segfaults. See `MapNet.forward` and `M2M.forward` in `lanegcn_onnx.py` for details.
***
## Quick Start (Deployment Only)
### 1. Clone with submodule
```sh
git clone --recursive https://gitee.com/junhuisirup/lanegcn_deployment.git DMLaneGCN
cd DMLaneGCN
# If already cloned without --recursive:
git submodule update --init --recursive
```
### 2. Install deployment dependencies
```sh
conda create --name lanegcn python=3.10
conda activate lanegcn
# PyTorch (adjust CUDA version to match your system)
conda install pytorch torchvision cudatoolkit=11.8 -c pytorch
# Argoverse API (required by the original data loader)
pip install git+https://github.com/argoai/argoverse-api.git
# Deployment-specific dependencies
pip install -r requirements.txt
pip install tensorrt pycuda polygraphy # TensorRT 10.x
```
> Training/testing dependencies (Horovod, etc.) are listed in the original [`LaneGCN/README.MD`](LaneGCN/README.MD).
### 3. Prepare a trained checkpoint
Follow the original repo to train or download the pretrained `36.000.ckpt`, then place it under `results/lanegcn/`. Data preparation is also documented in `LaneGCN/README.MD`.
### 4. Export ONNX
```sh
# From a trained checkpoint
python onnx_exporter.py --weight results/lanegcn/36.000.ckpt
# With --synthetic flag (no dataset needed, for quick testing)
python onnx_exporter.py --weight results/lanegcn/36.000.ckpt --synthetic
```
Produces `lanegcn.onnx` (with dynamic axes for variable actor/lane-node counts).
### 5. Build TensorRT Engine
```sh
# fp32
python trt_exporter.py --onnx lanegcn.onnx --output-dir trt_engines --precision fp32
# fp16
python trt_exporter.py --onnx lanegcn.onnx --output-dir trt_engines --precision fp16
```
Produces `trt_engines/lanegcn.engine`.
### 6. Run Inference & Visualize
```sh
python demo_inference.py --engine trt_engines/lanegcn.engine --seq 155
```
### 7. Benchmark (5 Backends)
```sh
# ORT CUDA requires CUDA 12.x libs from nvidia pip packages
# (container ships CUDA 13.0, incompatible with ORT 1.24)
NVLIB=$(python -c "import nvidia, os, glob; print(':'.join(glob.glob(os.path.dirname(nvidia.__file__)+'/*/lib')))")
export LD_LIBRARY_PATH="$NVLIB:$LD_LIBRARY_PATH"
python benchmark_all.py --seq 110 --iterations 100 --output benchmark_results.json
```
For single-backend ONNX Runtime profiling:
```sh
python onnx_inference.py --model lanegcn.onnx --provider CUDA --runs 100
```
### 8. Scene Selection (Demo Material)
```sh
# Generate a 10×6 grid of candidate scenes for visual inspection
python select_scenes.py --num 60 --output scene_candidates.png
# Pick 4-6 scene IDs from the grid, then generate a high-quality GIF for each:
python demo_inference.py --engine trt_engines_w8a32/lanegcn.engine --seq --output demo_scene_.gif --data val/data
```
### 9. Generate Figures
```sh
python generate_figures.py
```
Produces `misc/benchmark_comparison.png`, `misc/pipeline.png`, `misc/shape_input_before_after.png`, `misc/int8_sensitivity_heatmap.png`.
***
## Key Engineering Notes
1. **Shape input tensor elimination** — The `pre_mask` / `suc_mask` dynamic slicing is replaced with `torch.where(valid, ...)` zeroing. This removes all shape input tensors from the ONNX graph, working around a TensorRT 10.x `execute_async_v3` segfault. See [lanegcn\_onnx.py](lanegcn_onnx.py).
2. **Engine reuse** — `demo_inference.py` loads the TRT engine and persistent context once, then reuses them across inferences to avoid repeated deserialization overhead.
3. **`horovod`** **config** — Set to `False` in `lanegcn_onnx.py` (deployment variant). The training variant (`LaneGCN/lanegcn.py` in the submodule) keeps it `True` for distributed training.
4. **`MapQuery`** **map path** — Configurable via the `LANEGCN_MAP_DIR` environment variable (defaults to `./dataset/map_npy/`).
5. **Submodule + patched files** — `LaneGCN/` is a git submodule containing the original code. `data.py` and `layers.py` in the parent repo contain compatibility patches (`np.bool_`, `math.gcd`, `_build_cross_edges`). Deployment scripts configure `sys.path` so local patched files take precedence, then fall back to the submodule for unmodified modules (`utils.py`, etc.).
***
## License
Released under the [Uber Non-Commercial License](LICENSE) (© Uber Technologies, Inc.). Use is restricted to **non-commercial purposes** such as teaching, academic research, and personal experimentation. Commercial use is prohibited.
The ONNX / TensorRT deployment pipeline added by this repository is subject to the same license terms.
***