transfusion-p150 on Tenstorrent p150

Derived from AutowareFoundation/lidar_transfusion (tag v2.1), the TransFusion-L network that TIER IV trained for Autoware's autoware_lidar_transfusion LiDAR detector, with unmodified weights. TransFusion: Bai et al., arXiv:2203.11496. See the original model card for training details.

TransFusion-L (t4xx1_90m), the LiDAR-only TransFusion detector of Autoware's opt-in lidar_transfusion node: a PointPillars pillar encoder (11 decorated features, PFN 32-64) on a 768x768 pillar grid, a SECOND backbone, a SECONDFPN neck and a TransFusion head (dense heatmap, 3x3 local maximum, top-500 object queries, one transformer decoder layer over the 192x192 BEV, box heads). One LiDAR point cloud in base_link goes in (x, y, z and 0-255 intensity; one sweep, or Autoware's 2-sweep densified cloud). Out come 3D boxes of CAR, TRUCK, BUS, TRAILER, BICYCLE and PEDESTRIAN equivalent to Autoware's DetectedObjects, with the pre- and post-processing of the autoware_lidar_transfusion node (voxelization, decoration, decode, circle NMS, IoU NMS, area-based class remap).

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

LiDAR 3D object detection for Autoware-style sensor setups (concatenated clouds in base_link): research, prototyping and offline processing of recorded LiDAR data, with the outputs of Autoware's lidar_transfusion node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/transfusion-p150

tt serve downloads the Docker image and the AutowareFoundation/lidar_transfusion at 8f07eb93038bfd62e35d1f78e56a6b3fb76b12aa weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/transfusion-p150.

Send the sample point cloud shipped in this repo (Autoware's own test.pcd, as an npz) to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/transfusion-p150/resolve/main/code/tt_transfusion/samples/test_pcd.npz
printf '{"points": {"format": "npz", "data": "%s"}}' "$(base64 -w0 test_pcd.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one point cloud as JSON and returns its 3D detections. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

Response for the sample cloud (two of the five detections shown, meta shortened):

{"model": "transfusion-p150", "frame_id": "base_link", "num_detections": 5,
 "detections": [
   {"label": "TRUCK", "label_id": 2, "score": 0.3686, "center": [-19.688, 33.146, 1.187], "size": [6.132, 1.572, 1.868], "yaw": -2.8107},
   {"label": "TRUCK", "label_id": 2, "score": 0.1476, "center": [-34.117, 31.017, 1.918], "size": [7.078, 1.633, 2.0], "yaw": -0.7621}],
 "meta": {"variant": "default", "num_points": 48048, "num_pillars": 13058, "pillar_overflow": false, "sampling": {"mode": "clean", "offset": 0}, "counts": {"proposals": 500, "after_threshold": 6, "after_circle_nms": 6, "after_iou_nms": 5}},
 "timing_ms": {"preprocess": 5.539, "device": 23.802, "postprocess": 0.7, "total": 49.308, "decode": 19.009, "model_call": 30.066}}

• center, size ([length, width, height]) and yaw are in base_link metres and radians, in Autoware's convention for this node (z is the box centre, yaw counter-clockwise from +x); label_id is the autoware_perception_msgs ObjectClassification value; detections are sorted by score.
• No velocity is returned (Autoware ignores the network's velocity rows). TRAILER and remapped TRUCK labels come from Autoware's area-based class remapper; meta.label_before_remap keeps the network's class.
• A Python API with the same outputs (tt_transfusion.TransFusion) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Boxes computed by this package on 50 two-sweep LiDAR frames of Argoverse 2 Sensor val log 51bbdd4d (Miami, 10 Hz), in a 3D chase-camera view of the points, also as MP4. Non-commercial use only: rendered from the Argoverse 2 dataset, © 2021 Argo AI, LLC, CC BY-NC-SA 4.0 and the Argoverse Terms of Use; Argo AI does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing, identical voxel input); detections match when they have the same label and BEV centres at most 0.5 m apart:

Development frames: the sample cloud, 8 PandaSet, 5 nuScenes v1.0-mini mini_val, 3 Argoverse 2 val and 4 frames of the Autoware sample rosbag. Held-out frames were never used for a precision choice. No dataset metric (mAP, NDS) is reported: the weights were trained on TIER IV's internal data only, and the weights repo publishes no dataset-level accuracy to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations per stage; device time over 5 rounds of 50 back-to-back replays) and 50 served requests of the sample cloud. The model() and served times are host-bound: the host pre-processing of the cloud (loading, densifying, voxelization, decoration and pillar indexing) takes about 11 ms (one sweep) to 37 ms (two dense sweeps) of the call, and the server's base64 and npz decode of the request another 19 ms.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a TransFusion accuracy benchmark: the weights were trained on TIER IV's internal data only (Japanese taxis), and accuracy on other LiDAR setups is not characterized.
• Not a ROS 2 node: it serves HTTP and Python only, and takes arrays or files, not PointCloud2 messages. It runs the v2.1 (t4xx1_90m) model that Autoware's node pins; the other TransFusion-L releases of AWML are not included. In Autoware this detector is opt-in; CenterPoint is the default.
• Batch 1, one cloud per request; requests are served one at a time. Up to 65,536 pillars per frame on a fixed 768x768 grid; above that the first pillars in raster order are kept and meta.pillar_overflow is set.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Deterministic clean point sampling by default, where the node shuffles points with GPU atomics; autoware_compat emulates the shuffle but is not bit-exact. Points at exactly (0, 0, 0) and non-finite points are dropped. Outputs were compared with the fp32 CPU reference, not with Autoware's TensorRT engine.
• Mixed precision on the device (two-term bf16 weights and activations in the pillar net and the backbone, fp32 weights and bf16 activations elsewhere, fp32 accumulation): scores and boxes differ slightly from the fp32 reference. The 3x3 local maximum compares heat in bf16, so a heatmap peak that leads its neighbour by less than about 0.5 % can be missed or doubled (3 of 3,118 reference proposals missed, 22 extra in 55 frames), and an object near the 0.1 score threshold, the 0.3 rotation-head norm gate or the 12.1 / 36 m² remap areas can appear, disappear or change class.
• On public frames the weights themselves (on CPU and on p150 alike) often miss pedestrians in groups and standing or sitting people, do not detect parked bicycles, and turn 12-14 m buses into TRAILER through Autoware's remapper.

Licensing

• Weights: AutowareFoundation/lidar_transfusion at tag v2.1 (commit 8f07eb93038bfd62e35d1f78e56a6b3fb76b12aa), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. Trained by TIER IV on its internal data (not public).
• Pre- and post-processing ported from autoware_universe perception/autoware_lidar_transfusion (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample cloud code/tt_transfusion/samples/test_pcd.npz: Autoware's own test.pcd from autoware_universe perception/autoware_ground_segmentation/test/data/ (Apache-2.0).
• Demo media in media/ are renders, not part of the package. The demo video comes from Argoverse 2: non-commercial use only, CC BY-NC-SA 4.0, © 2021 Argo AI, LLC. The nuScenes renders (media/*_NC.*): non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). No dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.
• changh95/centerpoint-p150: Autoware's default LiDAR 3D detector.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/transfusion-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: