Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?

Paper | Project Page | GitHub Repository

Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving, yet it remains unclear when these representations improve driving performance. ViRA is a planner-agnostic visual representation alignment framework: during training it aligns planner features with a frozen VFM, while the deployed planner architecture and inference cost stay unchanged.

Findings

The study is organized around three empirical findings:

1. Consistent driving gains: VFM-guided visual representations improve driving across regression, diffusion, and scoring planners. The gains extend to zero-shot closed-loop evaluation.
2. Target selection matters: Different VFM targets yield different planning gains for the same planner. Alignment to another VFM can further benefit planners that already use a pre-trained VFM encoder.
3. Supervision changes sensitivity: Auxiliary perception supervision narrows the EPDMS spread across five VFM targets from 2.7 to 0.5 points, and can compensate for a less effective target.

Performance

Guided by these findings, ViRA-Diffusion aligns a diffusion planner to DINOv3 and is trained without auxiliary perception supervision. It reaches 92.3 EPDMS on NAVSIM v2 navtest, exceeding the recent methods in our comparison by at least 1.9 points.

The same alignment improves three camera-only planners. On NAVSIM v2 navtest, EPDMS rises by +5.4 for TransFuser (83.6 → 89.0), +7.4 for DiffusionDrive (84.5 → 91.9), and +10.9 for Rap* (72.8 → 83.7). On the challenging navhard split, Rap*-ViRA gains +16.4 EPDMS. Checkpoints trained only on NAVSIM also improve zero-shot closed-loop HD-Score on HUGSIM.

Camera-only. Rap* is our reimplementation with a different backbone and without the original data augmentation. HUGSIM is zero-shot: planners are trained only on NAVSIM. ViRA-Diffusion is DiffusionDrive trained with DINOv3 alignment and without auxiliary perception supervision. DiffusionDrive-ViRA (91.9) and ViRA-Diffusion (92.3) are distinct configurations.

License

All content in this repository is licensed under the Apache 2.0 license.

HUGSIM Closed-loop Visualization

(a) Car following: the lead vehicle brakes suddenly

(b) Oncoming traffic: other vehicles enter the current driving area

Citation

If you find ViRA useful in your research, please consider citing:

@article{zhang2026vira,
  title={Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?},
  author={Zhang, Zihao and Tian, Haochen and Li, Tianyu and Jing, Changhui and He, Jingliang and Ye, Naisheng and Pu, Ziyuan and Yang, Zhenjie},
  journal={arXiv preprint arXiv:2610.09695},
  year={2026},
  url={https://arxiv.org/abs/2610.09695}
}