bevfusion-p150 on Tenstorrent p150

Derived from AutowareFoundation/bevfusion (tag v2.0), the lidar-only BEVFusion-L network that TIER IV trained for Autoware's autoware_bevfusion LiDAR detector, with unmodified weights. BEVFusion: Liu et al., arXiv:2205.13542. See the original model card for training details.

BEVFusion-L, the lidar-only model of the weights repo (bevfusion_lidar.onnx, 8.3M parameters): hard voxelization (0.17 x 0.17 x 0.2 m voxels on a 1440 x 1440 x 41 grid) with a mean voxel encoder, a 21-layer sparse 3D convolution encoder to a 180 x 180 bird's-eye-view map, a SECOND backbone with a SECONDFPN neck and a TransFusion head (heatmap, top-500 object queries, one transformer decoder layer). It uses the LiDAR only: no camera input. One point cloud in base_link goes in (x, y, z; one sweep, or Autoware's two-sweep densified cloud with a time_lag column; intensity is not used). Out come Autoware DetectedObjects-equivalent 3D boxes for CAR, TRUCK, BUS, TRAILER, BICYCLE and PEDESTRIAN, with the pre- and post-processing of the autoware_bevfusion node (voxelization, two-sweep densification, score threshold, yaw-norm gate, circle and IoU NMS, area-based class remapper).

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

LiDAR 3D object detection for Autoware-style sensor setups (a concatenated point cloud in base_link): research, prototyping and offline processing of recorded drives, with the outputs of Autoware's autoware_bevfusion node in its lidar-only configuration.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/bevfusion-p150

tt serve downloads the Docker image and the AutowareFoundation/bevfusion at e1bf164b909d3c6c2642012f9b5ee4d827753220 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/bevfusion-p150.

Send the sample point cloud shipped in this repo (Autoware's own test.pcd, one sweep) to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/bevfusion-p150/resolve/main/code/tt_bevfusion/samples/test_pcd.npz
printf '{"points": {"format": "npz", "data": "%s"}}' "$(base64 -w0 test_pcd.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

code/tt_bevfusion/server/client.py in this repo (standard library only) builds the same request from .npy, .npz, .pcd or raw .bin clouds, with past sweeps.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one LiDAR point cloud as JSON and returns its 3D detections. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

Camera images and planner inputs are refused with a 400.

Response for the sample cloud (two of the five detections shown, meta shortened):

{"model": "bevfusion-p150", "frame_id": "base_link", "num_detections": 5,
 "detections": [
   {"label": "TRUCK", "label_id": 2, "score": 0.7584, "center": [24.184, 11.192, 1.535], "size": [8.107, 2.519, 2.849], "yaw": 3.1413},
   {"label": "TRUCK", "label_id": 2, "score": 0.5795, "center": [20.807, -0.494, 1.61], "size": [6.096, 2.339, 3.109], "yaw": -0.0158}],
 "meta": {"variant": "lidar", "num_voxels": 21359, "voxel_overflow": false, "sparse": {"bucket": "s", "overflow": false},
          "counts": {"proposals": 500, "after_threshold": 5, "after_circle_nms": 5, "after_iou_nms": 5}},
 "timing_ms": {"preprocess": 12.562, "device": 18.116, "postprocess": 0.911, "wait": 0.025, "total": 57.68, "stage": 1.145, "decode": 24.546, "model_call": 32.816}}

• center, size ([length, width, height]) and yaw are in base_link, metres and radians, Autoware's convention for this node: z is the box centre, yaw counter-clockwise from +x. label_id is the autoware_perception_msgs ObjectClassification value; score is Autoware's existence_probability. Detections are sorted by score.
• No velocity is returned (Autoware does not publish the network's velocity rows). TRAILER and remapped TRUCK labels come from Autoware's area-based class remapper (remap_classes: false turns it off).
• A Python API with the same outputs (tt_bevfusion.BEVFusion) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

3D boxes computed by this package from the LiDAR alone on 50 LIDAR_TOP sweeps of nuScenes scene-0103 (Boston Seaport, 10 Hz, two sweeps per frame), drawn from above; the CAM_FRONT image is an inset for orientation only. Also as MP4. Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing, same voxel input); a detection matches when the label is the same and the BEV centres are less than 0.2 m apart:

Public driving data: PandaSet (Pandar64) and nuScenes v1.0-mini (32-beam) frames, plus one frame of Autoware's sample rosbag (VLS-128); the held-out rows are frames never used to choose the device precision. The weights were trained on TIER IV's internal data only, and the weights repo publishes no dataset-level accuracy (mAP / NDS) to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (20 warm iterations per stage, two runs, ranges over the runs; device time over 5 rounds of 50 back-to-back replays) and 50 served requests of the sample cloud to the package's server app run on the host (two runs, mean over the runs). The model() and served times are host-bound: host pre-processing (voxelization and the sparse level sets) takes 14 ms of the sample call and 49 ms of the PandaSet call, and decoding the request takes 20.2 ms of the served time.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a BEVFusion accuracy benchmark: the weights are TIER IV's, trained on internal data that is not published, and accuracy on other LiDAR setups is not characterized.
• LiDAR only. The camera-lidar model of the weights repo (bevfusion_camera_lidar.onnx with its Swin-T image backbone) is not included, and the package refuses camera images. Not a ROS 2 node: it serves HTTP and Python only, and takes arrays or files instead of PointCloud2 messages.
• Batch 1, one cloud per request; the chip runs one cloud at a time. Up to three concurrent requests overlap their host stages with the device, but the container ships no TBB runtime, so the parallel host stages of concurrent requests run one at a time inside it.
• The sparse encoder runs in three fixed capacity buckets. A cloud whose active voxels per encoder level exceed the largest bucket (262,144 / 262,144 / 131,072 / 65,536 / 65,536 rows) is refused with a 400 error; nothing is clipped. With the level ratios of dense PandaSet clouds this can happen from about 180,000 voxels on, below Autoware's 256,000-voxel cap.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Each server start compiles the host stages, about 38 s even with a warm kernel cache.
• Differences from the deployed node: voxelization is deterministic (the first 10 points of each voxel in input order, and the first voxels in input order above the 256,000-voxel cap), and points at exactly (0, 0, 0) or with non-finite coordinates are dropped. Outputs were compared with the fp32 CPU reference, not with TensorRT.
• Device precision: fp32 weights with bf16 activations in most layers, two-term bf16 weights and activations with fp32 sums in the SECOND backbone. Scores differ from the fp32 reference by up to 0.014 on the evaluated clouds, so a detection near one of Autoware's decision thresholds (score 0.1, the rotation-norm gate 0.3, circle NMS 0.5 m, the class remapper's 12.1 / 36 m² areas) can appear, disappear or change label: on one rosbag frame two boxes that agree to 0.1 m switch between TRUCK and TRAILER at the 36 m² edge.
• On public driving data the weights themselves (on CPU and on p150 alike) lose range on 32-beam LiDAR (nuScenes car recall falls from 0.96 below 10 m to 0.24 at 40-50 m, against 0.80-1.00 up to 50 m on the 64-beam PandaSet), find ridden but not parked bicycles, and detect buses worst of all classes.

Licensing

• Weights: AutowareFoundation/bevfusion at tag v2.0 (commit e1bf164b909d3c6c2642012f9b5ee4d827753220), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. The upstream card names TIER IV's internal database as the training data; no public-dataset licence applies to the weights.
• Pre- and post-processing ported from autoware_universe perception/autoware_bevfusion (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample cloud code/tt_bevfusion/samples/test_pcd.npz: derived from autoware_universe's perception/autoware_ground_segmentation/test/data/test.pcd (Apache-2.0).
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.jpg come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/bevfusion-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: