sceneseg-p150 on Tenstorrent p150

Derived from SceneSeg, the scene segmentation network of the Autoware Foundation's VisionPilot camera stack (autowarefoundation/vision_pilot, formerly autoware.privately-owned-vehicles), as its scene_seg_model node deployed it, with unmodified weights: VisionPilot's SceneSeg_FP32.onnx, re-hosted from the upstream Google Drive link in the weights/ folder of this repo (the upstream card AutowareFoundation/SceneSeg holds no weights). SceneSeg has no paper. See the original model card for training details.

SceneSeg, a 48.3M-parameter camera segmentation network (EfficientNet-B0 encoder, a SceneContext block that gates the stride-32 features with a global prior, and a UNet-style decoder of five 2x up-blocks with a 3-class head) on a fixed 640x320 input (367 GFLOPs per frame). One camera image of any size goes in. Out come the VisionPilot foreground mask (mono8, 255 = foreground, at the input's size) and a 3-class map (background, foreground objects, drivable road), with the pre- and post-processing of the scene_seg_model node (squash to 640x320 with OpenCV's INTER_LINEAR, ImageNet normalisation, argmax, nearest-neighbour resize back).

Runs on Tenstorrent p150 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Camera scene segmentation (foreground objects and drivable road) for a VisionPilot-style front camera of 52-55° horizontal field of view: research, prototyping and offline processing of recorded driving images, with the output of VisionPilot's scene_seg_model node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/sceneseg-p150

tt serve downloads the Docker image and the changh95/sceneseg-p150 at a6829b246a05dba55c356323bbf75f7b1d148499 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/sceneseg-p150.

Send the sample image shipped in this repo (a VisionPilot tutorial image) to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/sceneseg-p150/resolve/main/code/tt_sceneseg/samples/highway_normal_1.png
printf '{"images": [{"camera": "CAM_FRONT", "data": "%s"}], "params": {"class_map": true}}' "$(base64 -w0 highway_normal_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Serving profiles

Select a different serving profile using --profile.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one camera image as JSON and returns its foreground mask and class map. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

Response for the sample image (bgr profile; the PNG data elided):

{"model": "sceneseg-p150", "frame_id": "camera",
 "meta": {"channel_order": "bgr", "source_hw": [414, 727], "network_hw": [320, 640], "foreground_class": 1, "mask_value": 255, "foreground_fraction": 0.040156},
 "mask": {"format": "png", "key": "mask", "dtype": "uint8", "shape": [414, 727], "data": "<base64 PNG>"},
 "class_names": ["background", "foreground", "road"],
 "class_counts": {"background": 166581, "foreground": 12086, "road": 122311},
 "class_map": {"format": "png", "key": "class_map", "dtype": "uint8", "shape": [414, 727], "data": "<base64 PNG>"},
 "timing_ms": {"preprocess": 1.976, "device": 13.236, "postprocess": 0.749, "total": 44.344, "decode": 18.425, "model_call": 15.978}}

• mask is what the VisionPilot node publishes on /autoseg/scene_seg/mask: one uint8 per pixel of the input image, 255 where the per-pixel argmax is class 1 (foreground), else 0, resized back from 640x320 with nearest neighbour.
• class_map is the 3-class argmax on the same pixel grid (0 background including sky, 1 foreground objects, 2 drivable road), which the node does not publish; class_counts counts its pixels.
• A Python API with the same outputs (tt_sceneseg.SceneSeg) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Foreground mask and road class computed by this package (bgr profile) on 50 front-camera images of Argoverse 2 Sensor val log 280269f9 (Pittsburgh, a downtown intersection, 10 Hz), as a 16:9 band of the portrait camera image, also as MP4. Non-commercial use only: rendered from the Argoverse 2 dataset, © 2021 Argo AI, LLC, CC BY-NC-SA 4.0 and the Argoverse Terms of Use; Argo AI does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (ONNX Runtime on the deployed SceneSeg_FP32.onnx, same pre- and post-processing), bgr profile unless stated:

Gate frames: the 9 VisionPilot tutorial images, 2 comma10k images, 8 PandaSet and 4 nuScenes frames. Held-out frames: KITTI 200, comma10k validation 100, nuScenes 55, PandaSet 48, comma10k 12. The ground-truth rows map the dataset labels to SceneSeg's three classes (vehicles, two-wheelers and people are foreground); the LiDAR rows are a sparse proxy (points 1-40 m projected into the camera) and the comma10k validation split is reconstructed from the upstream code. SceneSeg has no published benchmark to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations per stage; device time over 5 rounds of 50 back-to-back replays) and 50 (bgr) / 20 (rgb) served requests of the sample image. The model() and served times are host-bound: image decode takes about 11 ms of the call, and the server's base64 and PNG decode of the request another 17 ms.

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use it as an Autoware Universe component: SceneSeg is not part of Autoware Universe, and VisionPilot 1.0 no longer ships it. The package reproduces the scene_seg_model node of VisionPilot's deployment code at commit 04fa3e80.
• The default bgr profile reproduces the deployed node exactly, including its channel order: it feeds B, G, R planes to a network trained on R, G, B. Use rgb for better masks, above all on two-wheelers and people (6 % of PandaSet bicycle points come out foreground in bgr, 79 % in rgb); rgb is not what VisionPilot ran.
• Not a ROS 2 node: it serves HTTP and Python only. The INT8 QAT model and SceneSegLite (a different 19-class network) are not included.
• Batch 1, one image per request; requests are served one at a time. The network input is fixed at 640x320: any image is squashed to it without keeping the aspect ratio, as the node does.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Outputs were compared with the fp32 CPU reference (ONNX Runtime), not with VisionPilot's TensorRT FP16 engine.
• Mixed bf16 precision on the device (backbone weights and activations as two bf16 terms, a one-term bf16 decoder, fp32 accumulation): the class decision differs from the fp32 reference on a few pixels near class boundaries, at least 99.846 % of pixels equal on every held-out frame.
• The weights themselves (on CPU and on p150 alike) miss small and distant people and two-wheelers, label parking areas and about a third of sidewalk points as road, keep trams as background, and score lower on wider cameras (nuScenes, 65°) and on frames that show the ego hood.

Licensing

• Weights: weights/ of this repo at commit a6829b246a05dba55c356323bbf75f7b1d148499: VisionPilot's SceneSeg_FP32.onnx (sha256 8e509094…c973, checked at load) and its .pth as safetensors, re-hosted unchanged from the upstream Google Drive links with the upstream Apache-2.0 LICENSE, NOTICE and provenance. Downloaded by tt serve at that commit, not baked into the image. The upstream model card lists the training data (ACDC, MUSES, IDDAW, Mapillary Vistas, comma10K); check those datasets' terms before commercial use.
• Pre- and post-processing ported from autowarefoundation/vision_pilot VisionPilot/middleware_recipes/ROS2/models (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample images code/tt_sceneseg/samples/*.png: the upstream SceneSeg tutorial images (Apache-2.0). code/tt_sceneseg/samples/comma10k/: six comma10k images with their masks (MIT, © 2020 Comma.ai, Inc.).
• Demo media in media/ are renders, not part of the package. The demo video comes from Argoverse 2 (non-commercial use only, CC BY-NC-SA 4.0, © 2021 Argo AI, LLC); the nuScenes render media/sceneseg_nuscenes_scene0103_k28_front_tt_NC.jpg is non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc.; the KITTI render media/sceneseg_kitti_heldout_tt_NC.jpg is non-commercial use only, CC BY-NC-SA 3.0. The PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). No dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/sceneseg-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: