streampetr-p150 on Tenstorrent p150

Derived from AutowareFoundation/camera_streampetr (tag v1.0), the StreamPETR network that TIER IV trained for Autoware's autoware_camera_streampetr camera 3D detector, with unmodified weights. StreamPETR: Wang et al., arXiv:2303.11926. See the original model card for training details.

StreamPETR, an 82M-parameter camera-only 3D detector with a temporal object memory: a VoVNet-99-eSE + CPFPN image backbone on five cameras at 480x640, the PETR 3D position embedding (computed once per calibration) and a 6-layer transformer decoder over 644 learned and 256 propagated queries with a 1,024-row object memory kept on the device (about 1,100 GFLOPs per frame). Five surround-view camera images go in, with their intrinsics, camera-to-base_link extrinsics, the frame's time stamp and ego pose. Out come 3D boxes with scores and velocities for CAR, TRUCK, BUS, BICYCLE and PEDESTRIAN, with the pre- and post-processing of the autoware_camera_streampetr node (resize, crop and normalization, memory ego-motion compensation and resets, decoding from all six decoder layers, per-class score thresholds, IoU-BEV NMS).

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Camera-only 3D object detection with temporal fusion for Autoware-style five-camera rigs: research, prototyping and offline processing of recorded drives, with the outputs of one autoware_camera_streampetr node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/streampetr-p150

tt serve downloads the Docker image and the AutowareFoundation/camera_streampetr at 90f37d64186686ebc458509d69d3d4ca16d185c7 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/streampetr-p150.

Send the synthetic five-camera sample shipped in this repo (with its calibration preset) to /predict, on the port tt serve reported:

B=https://huggingface.co/changh95/streampetr-p150/resolve/main/code/tt_streampetr/samples/synthetic_5cam
CAMS="CAM_FRONT CAM_FRONT_LEFT CAM_BACK_LEFT CAM_FRONT_RIGHT CAM_BACK_RIGHT"
for c in $CAMS; do curl -sLO $B/$c.png; done
{ printf '{"calibration": {"preset": "synthetic_5cam"}, "images": ['; s=
  for c in $CAMS; do printf '%s{"camera": "%s", "data": "%s"}' "$s" $c "$(base64 -w0 $c.png)"; s=,; done
  printf ']}'; } > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

code/tt_streampetr/server/client.py in this repo (standard library only) builds requests from image files with a calibration file or preset and a stream id, time stamp and ego pose per frame.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one frame of the five cameras as JSON and returns its 3D detections. GET /health and GET /info report the server and device state and the calibration presets; GET /v1/models is a stub, not a chat API.

Response for the sample (three of the five detections shown, meta shortened):

{"model": "streampetr-p150", "frame_id": "base_link", "num_detections": 5,
 "detections": [
   {"label": "PEDESTRIAN", "label_id": 4, "score": 0.9715, "center": [9.334, -2.159, 0.65], "size": [0.803, 0.749, 1.813], "yaw": 2.3695, "velocity": [-0.003, 0.003]},
   {"label": "BUS", "label_id": 2, "score": 0.9697, "center": [8.977, 13.014, 1.606], "size": [11.762, 2.718, 3.189], "yaw": -2.8072, "velocity": [-0.001, -0.002]},
   {"label": "CAR", "label_id": 0, "score": 0.9669, "center": [11.932, 1.191, 0.554], "size": [4.186, 1.731, 1.701], "yaw": 2.9098, "velocity": [-0.001, 0.001]}],
 "meta": {"class_names": ["CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN"], "num_pre_nms": 30, "fresh": true, "skipped": null, "frame_index": 1},
 "timing_ms": {"preprocess": 20.058, "device": 38.295, "postprocess": 0.926, "total": 173.055, "decode": 113.471, "model_call": 59.32}}

• Boxes are in base_link, metres and radians, as the node publishes them: center is the box centre, size is [length along the heading, width, height] and yaw is counter-clockwise from +x. Detections are sorted by score.
• label is the class name, which the node also publishes as the Autoware label; label_id is the network class. velocity is the head's [vx, vy] in m/s, which the node computes but does not publish.
• meta carries the decoder layer and query of every box and the stream state (fresh, skipped, frame_index). A stream frame without an ego pose, or whose cameras are more than 0.15 s apart, returns no boxes and sets meta.skipped, where the node publishes nothing.
• A Python API with the same outputs (tt_streampetr.StreamPETR) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

3D boxes and 1.5 s track trails computed on the p150 from the five cameras of nuScenes scene-0796 (Singapore), run free-running over every CAM_FRONT frame (about 12 Hz) and shown at key-frames 1-20, with the front-camera strip. Also as MP4. The video was rendered with an earlier build of this port: same weights and precision, identical outputs on single frames and short streams; long free-running streams differ slightly (see Limitations). Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing). A reference box counts as recalled when a box of the same class lies within 1 m (F1 rows) and the reverse for precision; free-running means each side carries its own object memory through the whole sequence:

PandaSet 019 is a daytime San Francisco drive; nuScenes v1.0-mini scenes are from Boston and Singapore. Both are outside the weights' training domain (TIER IV's own five-camera data), so the mAP row is a cross-domain check of the port against the reference (nuScenes devkit, detection_cvpr_2019 settings on the 5 Autoware classes), not comparable with the paper. The weights repo publishes no dataset-level accuracy to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations for the blocking-frame and model() rows; the back-to-back row is the median over 3 batches of 30 trace replays) and 30 warm requests of the sample to the package's server (tt_streampetr.server.app, run with uvicorn on the host with the package's serve settings). The model() and served times are host-bound: resizing and normalizing the five images takes about 27 ms of the call, and decoding the 4.0 MB base64 PNG body takes 115.5 ms of the served time.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a StreamPETR accuracy benchmark: the weights are TIER IV's, trained on its own five-camera data, and accuracy on other rigs and public datasets is much lower (on CPU and p150 alike: vehicle recall / precision 0.37 / 0.40 on PandaSet 019 against the dataset's boxes).
• Not a ROS 2 node: it serves HTTP and Python only. Batch 1, one frame per request; requests are served one at a time. Fixed shapes: five cameras in the node's slots, a 480x640 network input, 644 + 256 queries, a 1,024-row memory and 5 classes. The server keeps one stream memory: a request with another stream id starts that stream from an empty memory.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Long free-running streams: the backbone's eSE channel-mean kernel adds its fp32 sums in a fixed order that differs from a plain matmul by about one fp32 ulp, and the temporal memory amplifies this. Single frames and short streams are unaffected, but over long streams the boxes drift further from the fp32 reference: free-running mean F1 0.912 on PandaSet 019, against 0.921 with the matmul order. Setting STREAMPETR_ESE_MEAN=matmul in the server environment selects the matmul order, at 2.4 ms more device time per frame.
• Device precision: fp32 weights, bf16 backbone activations, fp32 head activations, HiFi4 with fp32 accumulation. Scores differ from the fp32 reference by up to 0.032 on a single frame and more on warm stream frames, so an object near a class threshold can appear or disappear. The node's default fp16 TensorRT engines are not reproduced: outputs were compared with the fp32 CPU reference, not with TensorRT.
• Differences from the deployed node: the position embedding is recomputed when a request's calibration changes; skipped frames return an empty list with meta.skipped where the node publishes nothing; velocity is reported.

Licensing

• Weights: AutowareFoundation/camera_streampetr at tag v1.0 (commit 90f37d64186686ebc458509d69d3d4ca16d185c7), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. The upstream card names TIER IV's T4 dataset (AWML StreamPETR t4base v2.5, 123,708 frames) as the training data and documents no further data restriction.
• Pre- and post-processing ported from autoware_universe perception/autoware_camera_streampetr (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample code/tt_streampetr/samples/synthetic_5cam/ and its calibration preset: generated by this repo's code, no third-party data (Apache-2.0).
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.* come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/streampetr-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: