bevformer-p150 on Tenstorrent p150

Derived from changh95/bevformer-small-weights (commit 0e6d9d9), a re-hosting as safetensors of the official bevformer_small_epoch_24 checkpoint that the BEVFormer authors publish with fundamentalvision/BEVFormer: the BEVFormer-small network of Autoware's autoware_tensorrt_bevformer camera 3D detector, with unmodified weights. BEVFormer: Li et al., arXiv:2203.17270. See the original model card for training details.

BEVFormer-small, a 59.5M-parameter camera-only 3D detector (ResNet-101 with deformable convolutions in stages 3 and 4 and a one-level FPN, a 3-layer spatiotemporal BEV encoder over a 150x150 bird's-eye view that attends to the previous frame's BEV and to the six cameras, a 6-layer DETR decoder with 900 queries), run on six 1280x736 network inputs (about 1.97 TFLOPs per frame) through the graph of the TensorRT export Autoware deploys. Six surround-view camera images with their calibration and the ego pose go in. Out come 3D boxes with scores and velocities for the 10 nuScenes classes and the Autoware label of each, with the pre- and post-processing of the autoware_tensorrt_bevformer node (resize to 1600x900, normalisation, camera geometry, top-300 decode, label mapping). Each stream keeps its previous BEV on the device as temporal history.

Runs on Tenstorrent p150 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Camera-only 3D object detection for Autoware-style perception on a six-camera nuScenes-like rig: research, prototyping and offline processing of recorded drives, with the outputs of the autoware_tensorrt_bevformer node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/bevformer-p150

tt serve downloads the Docker image and the changh95/bevformer-small-weights at 0e6d9d9c7b8aca95820b264491e19a72d10f0608 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/bevformer-p150.

Send the synthetic six-camera sample shipped in this repo to /predict, on the port tt serve reported. client.py (standard-library Python) builds the request from the six images and the shipped calibration preset (--reset: a fresh BEV history, as in the response below):

hf download changh95/bevformer-p150 --include "code/tt_bevformer/*" --local-dir bevformer-p150 && cd bevformer-p150
S=code/tt_bevformer/samples/synthetic_6cam
python3 code/tt_bevformer/server/client.py --calib-preset synthetic_6cam --reset --out req.json \
  --image CAM_FRONT=$S/CAM_FRONT.png --image CAM_FRONT_RIGHT=$S/CAM_FRONT_RIGHT.png --image CAM_FRONT_LEFT=$S/CAM_FRONT_LEFT.png \
  --image CAM_BACK=$S/CAM_BACK.png --image CAM_BACK_LEFT=$S/CAM_BACK_LEFT.png --image CAM_BACK_RIGHT=$S/CAM_BACK_RIGHT.png
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Serving profiles

Select a different serving profile using --profile.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one frame of six camera images as JSON and returns its 3D boxes. GET /health and GET /info report the server and device state (/info lists the calibration presets); GET /v1/models is a stub, not a chat API.

Response for the sample with the default fast profile (three of the six detections shown, meta arrays elided):

{"model": "bevformer-p150", "frame_id": "LIDAR_TOP", "num_detections": 6,
 "detections": [
   {"label": "CAR", "label_id": 0, "score": 0.9785, "center": [-0.331, 11.844, -2.98], "size": [1.842, 4.218, 1.505], "yaw": -0.6172, "velocity": [-0.0, 0.004]},
   {"label": "CAR", "label_id": 0, "score": 0.9766, "center": [-3.585, 25.779, -3.873], "size": [2.065, 4.951, 1.866], "yaw": 0.3894, "velocity": [-0.0, 0.003]},
   {"label": "PEDESTRIAN", "label_id": 8, "score": 0.971, "center": [-3.468, 8.908, -2.803], "size": [0.776, 0.839, 1.8], "yaw": 1.5032, "velocity": [-0.001, -0.016]}],
 "meta": {"class_names": ["car", "truck", "construction_vehicle", "bus", "trailer", "barrier", "motorcycle", "bicycle", "pedestrian", "traffic_cone"],
          "box_convention": "autoware", "rot": ["..."], "query": ["..."], "stream": {"id": "default", "fresh": true, "use_prev_bev": 0.0}},
 "timing_ms": {"decode": 1911.3, "preprocess": 4046.9, "device": 57.3, "postprocess": 1.5, "total": 6018.1}}

• Boxes are in the network's nuScenes-style LIDAR_TOP frame (x right, y forward, about 1.84 m above the road on the default virtual mount), the frame the Autoware node publishes; center is the gravity centre in metres. By default size and yaw follow the node's message: size = [w, l, h] and yaw = the heading minus 90 degrees; params: {"physical_boxes": true} gives [l, w, h] and the heading of the length axis.
• label is the Autoware label (bus, trailer and construction_vehicle become TRUCK; barrier and traffic_cone UNKNOWN); label_id is the network class in meta.class_names. velocity is [vx, vy] in m/s.
• The timing above is the first request after a boot; later requests of the same sample take about 613 ms on the server (see Expected performance).
• A Python API with the same outputs (tt_bevformer.BEVFormer) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

3D boxes (score ≥ 0.4) computed on the p150 for 20 consecutive key-frames of nuScenes scene-0916 (Singapore, 2 Hz, shown at 4 frames/s), projected into the six cameras and drawn in the bird's-eye view, also as MP4. The video was rendered from the fp32-weight (HiFi4) outputs of this port's initial version, the precision of the accurate profile, not from the fast default. Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

The two serve profiles differ in precision. fast (the default) uses bf16 weights (bfp8 in the backbone's third stage), HiFi2 math and a bfp8 residual stream in the backbone; accurate (--profile accurate) uses fp32 weights and HiFi4 math. Both keep bf16 activations and fp32 accumulation. Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing):

fast is accepted on task-level agreement (|ΔmAP| and |ΔNDS| ≤ 0.02, recall and precision ≥ 0.90 on nuScenes mini_val, scenes 0103 and 0916, history reset at each scene start); it does not meet the per-stage PCC thresholds that accurate meets. A box counts as matched when a box of the same class lies within 1 m (nuScenes) or 0.5 m (PandaSet, the sample). The mini_val figures are indicative (3 of the 10 classes do not occur); the checkpoint's published full nuScenes val result is 0.370 mAP / 0.479 NDS, not re-measured here. Scene-1077 is a training scene of the weights, so its row measures agreement, not accuracy.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (p50 of 30 warm iterations) and, inside the container, 2,000 (fast) and 200 (accurate) served requests round-robin over the sample and three PandaSet frames. The served times are host-bound: decoding and pre-processing the base64 request take most of the call (the device part of a fast request is about 57 ms).

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification. No Autoware launch file starts the BEVFormer node by default.
• Do not use its outputs as a BEVFormer accuracy benchmark: the mini_val figures cover 81 key-frames of 2 scenes, and nuScenes mini_val is also the set on which the fast precision settings were chosen. The held-out checks are 3 PandaSet frames and 5 frames of a nuScenes training scene.
• Not a ROS 2 node: it serves HTTP and Python only. Autoware's own bevformer_small.onnx is not obtainable; the package runs the official checkpoint through a re-implementation of the TensorRT export graph that Autoware's file comes from. Outputs were compared with the fp32 CPU reference, not with the node's fp16 TensorRT engine.
• Batch 1, one frame per request; requests are served one at a time. Fixed shapes: six cameras in the nuScenes slots, 1280x736 network input, a 150x150 BEV grid (±51.2 m), 900 queries, up to 4 stream histories.
• A rig whose cameras each see more than 6,144 BEV cells is refused with HTTP 400; the Autoware node accepts any rig.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• fast (the default) is reduced precision: against the fp32 reference, scores move by up to 0.156 on held-out PandaSet frames (0.049 with accurate), so a box near the 0.2 score threshold appears or disappears more often. Use --profile accurate when per-box parity with the fp32 network matters.
• The nuScenes rows feed the pre-processed network input; the served path runs the same pre-processing on the device from the uint8 camera images, and matches it bit for bit in a kernel test.
• The weights were trained on nuScenes only (Boston, Singapore). Other rigs need a virtual LIDAR_TOP frame (the default calibration provides one) and lose accuracy, on CPU and p150 alike: on PandaSet sequence 019, frame 40, 22 of the 63 published boxes match one of the 78 annotations within 50 m. The network has no time-step input; send frames at about 2 Hz, the key-frame rate it was trained on.
• The first request after a boot takes about 6 s on the host, and the first frame of a new camera image size compiles a host resize kernel in the background (about 6 s, slower resizes until then).

Licensing

• Weights: changh95/bevformer-small-weights at commit 0e6d9d9c7b8aca95820b264491e19a72d10f0608 (branch main), Apache-2.0 per its model card: the tensors of the official bevformer_small_epoch_24.pth published by the BEVFormer authors (fundamentalvision/BEVFormer, Apache-2.0), unchanged. Downloaded by tt serve at that commit, not redistributed here. The weights were trained on nuScenes (CC BY-NC-SA 4.0; commercial use of nuScenes needs a licence from Motional): read the weights card's training-data notice before commercial use.
• Pre- and post-processing ported from autoware_universe perception/autoware_tensorrt_bevformer (Apache-2.0). The network graph follows DerryHub/BEVFormer_tensorrt (Apache-2.0); no code was copied from it.
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample code/tt_bevformer/samples/synthetic_6cam* and the calibration preset synthetic_6cam: generated by this package (code/scripts/make_sample.py, no third-party data), Apache-2.0.
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.* come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/bevformer-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: