ptv3-p150 on Tenstorrent p150

Derived from AutowareFoundation/ptv3 (tag v4.0), the Point Transformer V3 network that TIER IV trained for Autoware's autoware_ptv3 LiDAR segmentation and detection node, with unmodified weights. Point Transformer V3: Wu et al., arXiv:2312.10035. See the original model card for training details.

Point Transformer V3, a 34.0M-parameter LiDAR network: 0.12 m voxels serialized along two Morton curves, a 5-stage encoder (sparse-conv positional encoding, attention over patches of 512 voxels, grid pooling; about 0.36 TFLOPs on a 117,000-voxel frame), a segmentation decoder for 15 classes and a TransFusion-style BEV detection head (top-500 queries). One concatenated LiDAR cloud in base_link (x, y, z, intensity 0-255) goes in. Out come, per input point, a class, its probability, the normalised entropy, Autoware's PointCloudClassification value and the obstacle-cloud membership, and with the seg_det profile 3D boxes for 7 classes (CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN, TRAFFIC_CONE, BARRIER), with the pre- and post-processing of the autoware_ptv3 node (crop, voxelization, serialization, source reconstruction, score threshold, NMS, class remapping).

Runs on Tenstorrent p150 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

LiDAR semantic segmentation (Autoware's ML ground removal) and, with the seg_det profile, 3D object detection for Autoware-style concatenated clouds: research, prototyping and offline processing of recorded drives, with the outputs of the autoware_ptv3 node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150
• Time for the first start: on an empty kernel cache the server compiles the kernels of all 31 capacity buckets, about 40 minutes on an 8-vCPU host; later starts are ready in about 1 minute.

Quickstart

tt serve changh95/ptv3-p150

tt serve downloads the Docker image and the AutowareFoundation/ptv3 at 1673104843af3c740145ac74dc4af6b8a4502bf3 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/ptv3-p150.

Send Autoware's own test cloud, shipped in this repo, to /predict, on the port tt serve reported (serve with --profile seg_det to get boxes too):

curl -sLO https://huggingface.co/changh95/ptv3-p150/resolve/main/code/tt_ptv3/samples/test_pcd.npz
printf '{"points": {"format": "npz", "data": "%s"}}' "$(base64 -w0 test_pcd.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Serving profiles

Select a different serving profile using --profile.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one LiDAR cloud as JSON and returns a class for every point (and boxes with seg_det). GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

Response for the sample with seg_det (arrays are base64 npz, their data elided; class lists shortened):

{"model": "ptv3-p150", "frame_id": "base_link", "num_points": 48048,
 "labels": {"format": "npz", "key": "labels", "dtype": "uint8", "shape": [48048], "data": "..."},
 "scores": {"format": "npz", "key": "scores", "dtype": "float32", "shape": [48048], "data": "..."},
 "classification": {"...": "uint8 [48048]"}, "entropy": {"...": "float32 [48048]"}, "filter_keep": {"...": "bool [48048]"},
 "class_names": ["car", "truck", "bus", "bicycle", "pedestrian", "traffic_cone", "barrier", "debris", "drivable_flat", "non_drivable_flat", "vegetation", "building", "vertical_thin", "static_clutter", "noise"],
 "class_counts": {"truck": 64, "debris": 1, "drivable_flat": 12, "non_drivable_flat": 17486, "vegetation": 24122, "building": 1789, "255": 4574, "...": 0},
 "num_detections": 1,
 "detections": [{"label": "TRUCK", "label_id": 1, "score": 0.1264, "center": [22.332, -0.724, 1.461], "size": [9.35, 2.567, 3.605], "yaw": -0.0085, "velocity": [0.0, 0.0], "autoware_label": "TRUCK"}],
 "meta": {"frame": {"num_points": 48048, "num_unique_voxels": 27346, "...": "..."}, "neighbour_mode": "alias-first", "reconstruction": "full"},
 "timing_ms": {"decode": "...", "preprocess": "...", "device": "...", "postprocess": "...", "total": "..."}}

• One row per input point, in input order. Points outside the crop (x, y in [-122.88, 122.88) m, z in [-3, 5) m) get label 255 (INVALID), probability 0 and entropy NaN, and stay in the obstacle cloud.
• labels are the 15 PTv3 classes; classification is the node's mapping to PointCloudClassification; filter_keep is the membership of the node's filtered cloud (drivable_flat, non_drivable_flat and noise removed).
• detections (profile seg_det) are in base_link, metres and radians, after the node's score threshold, NMS and class remapper; center is the box centre; autoware_label is the published ObjectClassification (TRAFFIC_CONE and BARRIER become HAZARD). The seg profile returns no detections.
• A cloud with fewer than 16,000 voxels gets the node's empty output (meta.rejected) unless params.min_voxels is lowered.
• A Python API with the same outputs (tt_ptv3.PTv3) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Per-point classes (left) and 3D boxes (right) computed by this package (seg_det profile, min_voxels 0) on 50 LiDAR sweeps of nuScenes scene-1100 (night, pedestrians at a crosswalk, 10 Hz), bird's-eye view ±32 m, also as MP4. The video was rendered with this port's initial version (same weights). Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing, identical host index tables). Per-point outputs are identical in the two profiles; boxes come from seg_det only:

The 21 gate clouds are 12 PandaSet frames, 4 frames of the T4 sample dataset and 4 nuScenes mini_val key-frames (run with min_voxels 0), plus the PandaSet sweep of the speed table, which is the same cloud as one of the 12 PandaSet frames and so counts twice in the pooled numbers; the held-out frames are the other 77 frames of PandaSet sequence 019 (3 of their 10,072 matched boxes differ in score by more than 0.05). The weights were trained on TIER IV's T4 dataset with 15 / 7 classes that public datasets label differently, so no dataset metric is reported; the weights repo publishes none.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (50 warm iterations per stage) and, inside the container, 350 served requests per input and profile over 5 container starts. Device time grows with the cloud: each cloud runs in the smallest of 31 capacity buckets that holds it. The model() and served times are host-bound: pre-processing and index tables take about 30 ms of a test.pcd call, and the PandaSet sweep (106,501 points) carries more.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification. In Autoware, PTv3 is an opt-in segmentation / ML ground removal that is off by default, and its detection head is idle; no tagged Autoware release runs these v4.0 weights.
• Do not use its outputs as a PTv3 accuracy benchmark or on other LiDAR setups without checking: the weights are TIER IV's training on about 4,000 frames of the non-public T4 dataset, and accuracy on other sensors and domains is not characterized.
• Not a ROS 2 node: it serves HTTP and Python only. One concatenated cloud per request: no multi-sweep, camera or stream input (the v4.0 node has none).
• Batch 1, one cloud per request; requests are served one at a time; one profile per server. At most 256,000 voxels per cloud (the node's maximum).
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Outputs were compared with the fp32 CPU reference, not with the node's TensorRT engines (fp16 by default). The sparse-conv neighbour tables reproduce spconv's key aliasing deterministically (alias-first, the default) or use exact keys; the engine's GPU hash order is not reproduced bit for bit.
• Outputs differ slightly from the fp32 reference: 0.1-0.8 % of point labels; boxes near the 0.1 score threshold, and NMS or top-k near-ties, can appear or disappear. On held-out PandaSet frames a matched box's score can differ by up to 0.17.
• Clouds below the node's 16,000-voxel minimum (every single-sweep nuScenes cloud) get the node's empty output unless min_voxels is lowered.
• On public LiDARs the weights themselves (on CPU and on p150 alike) do not transfer T4's fine static classes and give no traffic-cone or barrier detections.

Licensing

• Weights: AutowareFoundation/ptv3 at tag v4.0 (commit 1673104843af3c740145ac74dc4af6b8a4502bf3), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. Trained by TIER IV on the T4 dataset (about 4,000 frames); the training configuration is not public.
• Pre- and post-processing ported from autoware_universe perception/autoware_ptv3 (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample code/tt_ptv3/samples/test_pcd.npz: the test.pcd of the Autoware package (Apache-2.0), converted to npz.
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.* come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/ptv3-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: