Point Transformer V3 for Autoware (ptv3)

Dual-task 3D perception models for LiDAR point clouds, used by the autoware_ptv3 node in Autoware.

The models follow the Point Transformer V3 (PTv3) [1] architecture: a shared sparse-convolution point transformer encoder feeding two task heads, one for 3D object detection and one for 3D semantic segmentation. The sparse convolution backend corresponds to spconv, which Autoware consumes through spconv_cpp. The models are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from the ONNX files on first launch.

Model overview

The network is split into three ONNX sub-models, matching how the node consumes them:

1. Encoder (ptv3_encoder.onnx): shared PTv3 backbone with encoder channels [32, 64, 128, 256, 512], serialization orders z and z-trans, and pooling strides [2, 2, 2, 2].
2. Detection head (ptv3_det3d_head.onnx): predicts up to 500 oriented 3D boxes with velocity (has_twist: true) over a BEV grid with bbox_voxel_size: [0.96, 0.96, 8.0].
3. Segmentation head (ptv3_seg3d_head.onnx): per-point classification over the 15 segmentation classes listed above.

Each head can be enabled or disabled independently at launch time (use_det3d_head, use_seg3d_head). Pre-processing (voxelization) and post-processing (IoU NMS, yaw normalization, distance-based score thresholding, class remapping by area, point filtering) run in the node, not in the ONNX graphs.

Files

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Input: ~/input/pointcloud (sensor_msgs/msg/PointCloud2), by default remapped to /sensing/lidar/concatenated/pointcloud. The node detects the input point format automatically on the first received message and supports XYZIRCAEDT (10 fields), XYZIRADRT (9 fields), XYZIRC (6 fields), and XYZI (4 fields).

Outputs:

The node also publishes processing-time and latency debug topics under debug/.

Usage in Autoware

Autoware's setup (the Ansible artifacts role) downloads these artifacts to ~/autoware_data/ml_models/ptv3/, and the node launches with, e.g.:

ros2 launch autoware_ptv3 ptv3.launch.xml \
  model_path:=$HOME/autoware_data/ml_models/ptv3

Add build_only:=true to build the TensorRT engines from the ONNX files as a one-off pre-task. The ml_package_ptv3_*.param.yaml files in this repository are loaded from the model directory at launch, while node parameters (precision, workspace sizes, filtering, score thresholds) come from the package's config/ptv3.param.yaml. See the package README for the full parameter reference.

Training

The model was trained on the T4Dataset using approximately 4,000 frames, as documented in the package README. A public training configuration is not documented in the package.

• Sparse convolution backend: https://github.com/traveller59/spconv
• C++ sparse convolution implementation used by Autoware: https://github.com/autowarefoundation/spconv_cpp
• Point Transformer V3 paper: arXiv:2312.10035

Provenance

Limitations

• Trained on the T4Dataset sensor configuration; accuracy on a different LiDAR setup (mounting position, beam count, concatenated clouds) can drop without fine-tuning.
• Only the classes listed above are detected or segmented. Other road users fall outside the label set.
• Evaluation metrics for this model are not publicly documented.

Citation

@inproceedings{wu2024ptv3,
  title     = {Point Transformer V3: Simpler, Faster, Stronger},
  author    = {Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang},
  booktitle = {CVPR},
  year      = {2024}
}

References

• [1] Wu et al., "Point Transformer V3: Simpler, Faster, Stronger", CVPR 2024, arXiv:2312.10035.
• spconv: https://github.com/traveller59/spconv
• spconv_cpp: https://github.com/autowarefoundation/spconv_cpp