SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models
Changhui Jing, Haochen Tian, Hongchen Li, Tianyu Li, Caojun Wang, Zihao Zhang, Yanhao Liu, Chengen Xie, Bailin Li, Hao Zhao, Zhenjie Yang
Highlights
- 🚗 Simulation-grounded alignment: Render matched future videos for alternative candidate trajectories in reconstructed real-world scenes.
- 🔍 Selective video supervision: Prioritize higher-error futures among high-reward candidates and jointly train trajectory planning and video prediction.
- 📈 Improved planning: Achieve 91.3 PDMS on NAVSIM v1, 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM, with direct trajectory prediction at inference.
Overview
SimVTA is a simulation-grounded post-training framework for driving world action models. It extends supervision beyond logged trajectories using a 3D Gaussian Splatting simulator, combining trajectory GRPO with a flow-matching video loss that updates shared representations.
Reported results in the paper:
| Benchmark | Metric | SimVTA |
|---|---|---|
| NAVSIM v1 | PDMS | 91.3 (+5.1 over the 86.2 initialization) |
| NAVSIM v2 | EPDMS | 88.2 |
| HUGSIM | Average HD-Score | 38.8 |
Matched Futures for Alternative Actions
Simulator-rendered futures for going straight (A), changing lanes left (B), and changing lanes right (C).
Video-Trajectory Alignment
Video-trajectory alignment: World Engine renderings compared with SimVTA and Epona predictions.
Checkpoint Download
This repository provides simvta.pkl, the SimVTA checkpoint post-trained from a NAVSIM-adapted Epona initialization.
pip install -U huggingface_hub
hf download OpenDriveLab-org/SimVTA simvta.pkl --local-dir ./ckpts
The checkpoint will be saved to ./ckpts/simvta.pkl. Code and detailed inference instructions are being prepared for release in the GitHub repository. Additional pretrained components and dataset preparation will be documented there.
Acknowledgements
We thank the authors and open-source contributors of the following projects for providing the models, simulation tools, and evaluation benchmarks that support this work:
- Epona | World Engine | NAVSIM | HUGSIM
We also thank the broader open-source community for its contributions to driving world models, autonomous driving, and 3D Gaussian Splatting.
License and Citation
The SimVTA release is licensed under Apache-2.0, subject to the retained upstream MIT notice for Epona in THIRD_PARTY_NOTICES.md.
If our paper or checkpoint helps your research, please consider citing our work and giving the GitHub repository a star.