pointpainting-p150 on Tenstorrent p150

Derived from AutowareFoundation/image_projection_based_fusion (tag v5.0), the PointPainting network that TIER IV trained for Autoware's autoware_image_projection_based_fusion camera-LiDAR detector (pointpainting_fusion node), with unmodified weights. PointPainting: Vora et al., arXiv:1911.10150. See the original model card for training details.

PointPainting, a camera-LiDAR 3D object detector: the 2-D detections (ROIs) of up to 8 rectified cameras are projected onto the LiDAR points as 3 additive class bits, then a PointPillars pillar encoder (PFN 12-16-32), a SECOND backbone, a SECONDFPN neck and a single-task CenterPoint head on a 760x480 grid detect the objects. One LiDAR point cloud in base_link goes in, with the cameras' calibration and their 2-D detections (one sweep, client-side past sweeps or server-side 2-sweep densification). Out come 3D boxes of CAR, TRUCK, BUS, TRAILER, BICYCLE and PEDESTRIAN equivalent to Autoware's DetectedObjects, with the pre- and post-processing of the pointpainting_fusion node (painting, densification, voxelization, decode, circle NMS, IoU NMS, area-based class remap). The 2-D detector itself is not included.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Camera-LiDAR 3D object detection for Autoware-style sensor setups (concatenated LiDAR clouds in base_link plus the 2-D detections of up to 8 rectified cameras): research, prototyping and offline processing of recorded driving data, with the outputs of Autoware's pointpainting_fusion node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/pointpainting-p150

tt serve downloads the Docker image and the AutowareFoundation/image_projection_based_fusion at 33119d71afd53fbc3994dcffc71e6247b76cdd5a weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/pointpainting-p150.

Send the sample shipped in this repo (a synthetic 6-camera scene: two LiDAR sweeps, the calibration and 53 ROIs per sweep) to /predict, on the port tt serve reported. client.py (standard library only) turns the sample manifest into one request:

hf download changh95/pointpainting-p150 --include "code/tt_pointpainting/samples/*" "code/tt_pointpainting/server/*" "code/tt_pointpainting/ttaw/server/*" --local-dir pointpainting-p150 && cd pointpainting-p150
python3 code/tt_pointpainting/server/client.py --sample code/tt_pointpainting/samples/synthetic_6cam.json --out req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one point cloud with the cameras' calibration and 2-D detections as JSON and returns the 3D detections. GET /health and GET /info report the server and device state (/info lists the calibration presets and every parameter default); GET /v1/models is a stub, not a chat API.

Response for the sample (two of the 36 detections shown, meta shortened):

{"model": "pointpainting-p150", "frame_id": "base_link", "num_detections": 36,
 "detections": [
   {"label": "CAR", "label_id": 1, "score": 0.8622, "center": [31.202, 5.296, 0.812], "size": [4.87, 1.973, 1.719], "yaw": -0.0305},
   {"label": "CAR", "label_id": 1, "score": 0.7739, "center": [-27.343, -1.851, 0.867], "size": [4.855, 2.03, 1.84], "yaw": -0.0214}],
 "meta": {"variant": "default", "num_points": 219438, "num_pillars": 14213, "pillar_overflow": false, "counts": {"after_threshold": 137, "after_circle_nms": 51, "after_iou_nms": 36}, "painting": {"...": "points in view and painted, per camera"}},
 "timing_ms": {"preprocess": 15.186, "device": 18.732, "postprocess": 2.732, "total": 119.393, "decode": 81.531, "model_call": 36.699}}

• center, size ([length, width, height]) and yaw are in base_link metres and radians, in Autoware's convention; label_id is the autoware_perception_msgs ObjectClassification value; detections are sorted by score.
• No velocity is returned (the network has no twist head, as in Autoware). TRAILER and remapped TRUCK labels come from Autoware's area-based class remapper; meta.label_before_remap keeps the network's class.
• A Python API with the same outputs (tt_pointpainting.PointPainting) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Boxes computed by this package on 50 two-sweep frames of PandaSet sequence 065 (San Francisco, Embarcadero at night, 10 Hz), with the ROIs of a YOLOX CPU reference on the six cameras: a bird's-eye view with every point coloured by its paint bits, and three of the camera images with their ROIs. Also as MP4. Contains data from PandaSet (Scale AI and Hesai), https://pandaset.org, licensed under CC BY 4.0 and the PandaSet Dataset Terms; Scale AI and Hesai do not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same painting, pre- and post-processing, identical voxel input); detections match when they have the same label and BEV centres at most 0.5 m apart:

Held-out frames: PandaSet sequences 005 and 019 and nuScenes v1.0-mini mini_val, never used for a precision choice; their ROIs come from a CPU reference of Autoware's YOLOX model. No dataset metric (mAP, NDS) is reported: the weights were trained by TIER IV on undisclosed data with Autoware's classes, and the weights repo publishes no dataset-level accuracy to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations per stage; device time over 5 rounds of 50 back-to-back replays) and 50 served requests of the sample. The model() and served times are host-bound: painting, densification and voxelization take 15-20 ms of the call, and the server's base64 and npz decode of the 3.2 MB request 81.6 ms of the served time.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a PointPainting accuracy benchmark: the weights are TIER IV's, trained on undisclosed data, and accuracy depends on the 2-D detector that supplies the ROIs and on the camera-LiDAR calibration and time offsets. The shipped sample is synthetic and says nothing about accuracy on real data.
• The 2-D detector is not included: the ROIs are an input, and their time synchronisation with the cloud is the caller's job. In Autoware this detector runs only from the camera-LiDAR perception launch files; the default launch uses CenterPoint.
• Not a ROS 2 node: it serves HTTP and Python only, and takes arrays or files, not PointCloud2, CameraInfo or ROI messages. Only rectified pinhole cameras; the node's unrectified and approximate projection modes are not included.
• Batch 1, one frame per request; requests are served one at a time. At most 8 cameras, 128 ROIs per camera and 40,000 pillars on a fixed 760x480 grid; above 40,000 pillars the lowest raster cells are kept.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Autoware's additive paint codes are kept: two overlapping car boxes set the BICYCLE bit, exactly as in the node. Deterministic voxelization (the first 32 points of a pillar in buffer order) replaces the order of Autoware's CUDA atomics, and the score sort before circle NMS is stable. Outputs were compared with the fp32 CPU reference, not with Autoware's TensorRT engine.
• fp32 weights and bf16 activations: scores and boxes differ slightly from the fp32 reference. An object near Autoware's decision thresholds (score 0.35, rotation-head norm 0.3, the circle-NMS neighbourhood, the remap areas) can appear, disappear or come from a neighbouring cell: 23 of 2,292 reference detections in 31 held-out frames have no partner.
• On public frames the weights themselves (on CPU and on p150 alike) miss part of the pedestrians and of the vehicles on other LiDAR setups (72 % of the pedestrians on PandaSet 019, 71-77 % of the vehicles on nuScenes), and Autoware's remapper turns long buses into TRAILER.

Licensing

• Weights: AutowareFoundation/image_projection_based_fusion at tag v5.0 (commit 33119d71afd53fbc3994dcffc71e6247b76cdd5a), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. The upstream card does not disclose the training data (TIER IV).
• Pre- and post-processing ported from autoware_universe perception/autoware_image_projection_based_fusion and the autoware_lidar_centerpoint code it reuses (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample code/tt_pointpainting/samples/synthetic_6cam.*: a synthetic scene generated by code/scripts/make_sample.py, no dataset involved (Apache-2.0).
• Demo media in media/ are renders, not part of the package. The demo video and the PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). The nuScenes renders (media/*_NC.jpg): non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.
• changh95/yolox-p150: Autoware's 2-D camera detector, the usual source of the ROIs. changh95/centerpoint-p150: the LiDAR-only detector with the same backbone and head design.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/pointpainting-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: