bevdet-p150 on Tenstorrent p150

Derived from AutowareFoundation/tensorrt_bevdet (tag v1.0), the BEVDet-R50-4DLongterm-Depth network deployed by Autoware's autoware_tensorrt_bevdet camera 3D detector, with unmodified weights. BEVDet: Huang et al., arXiv:2112.11790; BEVDet4D: arXiv:2203.17054. See the original model card for training details.

BEVDet-R50-4DLongterm-Depth, a 76M-parameter camera-only 3D detector with temporal fusion: a ResNet-50 + FPN image encoder with a camera-aware depth network (118 depth bins) on six cameras at 256x704, a BEVPool v2 lift-splat into a 128 x 128 bird's-eye-view grid (±51.2 m), fusion of the last 8 BEV maps, a ResNet + FPN BEV encoder and a CenterPoint-style head (498 GFLOPs per frame). Six surround-view camera images go in, with their intrinsics and camera-to-base_link extrinsics (and optionally the ego pose for motion-compensated fusion). Out come 3D boxes with scores and velocities for the 10 nuScenes classes and the Autoware label of each, with the pre- and post-processing of the autoware_tensorrt_bevdet node (resize and crop, BEV history, box decoding, Scale-NMS, label mapping). The image and BEV encoders run on the device; the lift-splat between them, the image resize and the NMS run on the host.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Camera-only 3D object detection for Autoware-style six-camera rigs (Autoware's camera perception mode): research, prototyping and offline processing of recorded drives, with the outputs of one autoware_tensorrt_bevdet node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/bevdet-p150

tt serve downloads the Docker image and the AutowareFoundation/tensorrt_bevdet at 391466349a1b39c93b9865364a8088110ca8d97b weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/bevdet-p150.

Send the synthetic six-camera sample shipped in this repo (with its calibration preset) to /predict, on the port tt serve reported:

B=https://huggingface.co/changh95/bevdet-p150/resolve/main/code/tt_bevdet/samples/synthetic_6cam
CAMS="CAM_FRONT_LEFT CAM_FRONT CAM_FRONT_RIGHT CAM_BACK_LEFT CAM_BACK CAM_BACK_RIGHT"
for c in $CAMS; do curl -sLO $B/$c.png; done
{ printf '{"calibration": {"preset": "synthetic_6cam"}, "images": ['; s=
  for c in $CAMS; do printf '%s{"camera": "%s", "data": "%s"}' "$s" $c "$(base64 -w0 $c.png)"; s=,; done
  printf ']}'; } > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

code/tt_bevdet/server/client.py in this repo (standard library only) builds requests from image files with a calibration file or preset and a stream id and ego pose per frame.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one frame of the six cameras as JSON and returns its 3D detections. GET /health and GET /info report the server and device state, the labels and the calibration presets; GET /v1/models is a stub, not a chat API.

Response for the sample, the first request after the server starts (two of the ten detections shown, meta shortened). It fuses an empty history and builds the lift-splat tables for the new calibration, so its model_call takes 0.5 s; warm requests take the times under Expected performance, and repeated requests on the same stream fuse the BEV history, so their scores change slightly:

{"model": "bevdet-p150", "frame_id": "base_link", "num_detections": 10,
 "detections": [
   {"label": "traffic_cone", "label_id": 9, "score": 0.9572, "center": [7.639, -3.562, 0.298], "size": [0.376, 0.365, 0.78], "yaw": 0.128, "velocity": [-0.0, 0.0], "autoware_label": "UNKNOWN", "twist": {"linear_x": 0.0, "angular_z": 5.3139}},
   {"label": "car", "label_id": 0, "score": 0.924, "center": [6.039, 9.203, 0.873], "size": [4.827, 1.982, 1.967], "yaw": 0.2323, "velocity": [-0.0, -0.0], "autoware_label": "CAR", "twist": {"linear_x": 0.0, "angular_z": -4.2956}}],
 "meta": {"class_names": ["car", "truck", "..."], "autoware_labels": ["CAR", "TRUCK", "..."], "score_threshold": 0.2, "stream": {"id": "default", "frame": 0, "flag": 1}},
 "timing_ms": {"encode": 10.483, "bevpool": 8.029, "bev_decode": 8.796, "preprocess": 481.679, "device": 27.95, "postprocess": 1.834, "total": 935.998, "decode": 423.309, "model_call": 511.62}}

• center (z at the box's gravity centre), size ([length, width, height]) and yaw (counter-clockwise from +x) are in base_link, metres and radians, as the node publishes them. Detections are sorted by score.
• label is the network class (one of the 10 nuScenes classes, label_id its index); autoware_label is the label the node publishes (construction_vehicle, barrier and traffic_cone become UNKNOWN).
• velocity is [vx, vy] in m/s in base_link axes; twist is the node's twist.linear.x / twist.angular.z, reproduced as deployed.
• A Python API with the same outputs (tt_bevdet.BEVDet) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

3D boxes (score ≥ 0.4) computed on the p150 for 20 consecutive key-frames of nuScenes scene-0553 (Boston: waiting at an intersection, a cyclist, a large truck, pedestrians on the crosswalk; 2 Hz, shown at 4 frames/s), on the three front cameras and in bird's-eye view. Also as MP4. The video was rendered with an earlier build of this port (same weights and precision); its boxes differ from this package's only within the agreement figures under Expected performance. Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing). A reference box with score ≥ 0.2 counts as recalled when a box of the same class lies within 0.5 m, and the reverse for precision:

nuScenes rows: the 81 key-frames of the two mini_val scenes (Boston and Singapore; 3 of the 10 classes do not occur), scored with the nuScenes devkit (detection_cvpr_2019); the weights were trained on nuScenes, and the BEVDet authors report 0.394 / 0.515 mAP / NDS on the full validation set for this configuration, which 81 frames cannot reproduce. PandaSet 019 is a daytime San Francisco drive from another camera rig.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations per stage; the ranges span two sessions) and 40 requests of the sample to the package's server app running on the host. About two thirds of a model() call is host work: pre-processing with the image resize (about 3.5 ms at 1600x900, 7.3-7.5 ms at 1920x1080), the lift-splat between the two device stages (7-8 ms) and box decoding with NMS (1.5-4.5 ms). Decoding the six base64 PNGs takes most of the served time.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a BEVDet accuracy benchmark: the mini_val figures cover 81 frames, and on camera rigs other than nuScenes' the weights' depth prior is biased (on CPU and p150 alike: on PandaSet front objects come out about 15 % too close and side objects 10-20 % too far).
• Not a ROS 2 node: it serves HTTP and Python only. Batch 1, one frame per request; requests are served one at a time. Fixed shapes: six cameras, a 256x704 network input, a 128 x 128 BEV grid (±51.2 m), 8 history frames and 10 classes; a rig whose lift-splat tables exceed 370,000 points / 14,000 intervals is refused, as in the TensorRT node. At most 4 stream histories are kept; a fifth stream id evicts the least recently used one.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• The first call on a new calibration or image size builds its lift-splat tables: 0.18-1.6 s, measured.
• By default the temporal fusion follows the deployed node (no ego-motion alignment, which lowers mAP / NDS by 0.11 / 0.16 on nuScenes mini_val); pass stream.T_world_from_ego for the motion-compensated history.
• Intrinsics are taken for the image as sent and scaled with the resize to 1600x900; the deployed node keeps the raw CameraInfo.k when it resizes (calibration.scale_intrinsics: false reproduces it, with far fewer and misplaced boxes on a 1920x1080 rig).
• Device precision: fp32 weights, bf16 activations, HiFi4 with fp32 accumulation. Scores differ from the fp32 reference by up to about 0.009, so an object within a few thousandths of the 0.2 publication threshold can appear or disappear. The node's default fp16 TensorRT engine is not reproduced: outputs were compared with the fp32 CPU reference, not with TensorRT.

Licensing

• Weights: AutowareFoundation/tensorrt_bevdet at tag v1.0 (commit 391466349a1b39c93b9865364a8088110ca8d97b), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. The upstream card's Legal Notice: the weights were trained on nuScenes (CC BY-NC-SA 4.0, non-commercial); read it before any commercial use.
• Pre- and post-processing ported from autoware_universe perception/autoware_tensorrt_bevdet (Apache-2.0). The node's inference library bevdet_vendor carries no licence; its behaviour was re-implemented from a written description, without copying code.
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample code/tt_bevdet/samples/synthetic_6cam/ and its calibration preset: generated by this repo's code, no third-party data (Apache-2.0).
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.* come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/bevdet-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: