frnet-p150 on Tenstorrent p150

Derived from AutowareFoundation/lidar_frnet (tag v2.0), the FRNet networks that TIER IV trained with AWML for Autoware's autoware_lidar_frnet LiDAR segmentation node, with unmodified weights. FRNet: Xu et al., arXiv:2312.04484. See the original model card for training details.

FRNet, a 10.0M-parameter LiDAR semantic-segmentation network (a frustum feature encoder of point-wise MLPs with a per-cell max, a frustum-range 2D CNN with four residual stages and point-pixel fusion, and a per-point head), one network per sensor: the Hesai OT128 on a 128x1024 frustum grid (about 1.25 TFLOPs per sweep at 80,000 points) and the Hesai QT128 on 128x256 (about 0.36 TFLOPs at 60,000 points), selected by serve profile. One LiDAR sweep in the sensor frame (x, y, z, raw 0-255 intensity) goes in. Out come a class id and its probability for every point (27 T4 classes) and the node's filtered-cloud mask, with the pre- and post-processing of the autoware_lidar_frnet node (ego crop box, frustum projection, Autoware's point interpolation, argmax, class filter).

Runs on Tenstorrent p150 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

LiDAR 3D semantic segmentation for Autoware-style perception with a Hesai OT128 or QT128 sensor: research, prototyping and offline processing of recorded sweeps, with the outputs of the autoware_lidar_frnet node.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/frnet-p150

tt serve downloads the Docker image and the AutowareFoundation/lidar_frnet at a53a1c11b0b28d31fd50a03abef5db0e830a5b3c weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/frnet-p150.

Send the synthetic OT128 sweep shipped in this repo to /predict, on the port tt serve reported (for the qt128 profile use synthetic_qt128.npz and the preset synthetic_qt128):

curl -sLO https://huggingface.co/changh95/frnet-p150/resolve/main/code/tt_frnet/samples/synthetic_ot128.npz
printf '{"points": {"format": "npz", "data": "%s"}, "calibration": {"preset": "synthetic_ot128"}}' "$(base64 -w0 synthetic_ot128.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Serving profiles

Select a different serving profile using --profile.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one LiDAR sweep as JSON and returns a class and probability for every point. GET /health and GET /info report the server and device state (/info lists the calibration presets); GET /v1/models is a stub, not a chat API.

Response for the OT128 sample (arrays are base64 npz, their data elided; class lists shortened):

{"model": "frnet-p150", "frame_id": "sensor", "num_points": 52613, "num_points_raw": 52613,
 "labels": {"format": "npz", "key": "labels", "dtype": "uint8", "shape": [52613], "data": "..."},
 "scores": {"format": "npz", "key": "scores", "dtype": "float32", "shape": [52613], "data": "..."},
 "points": {"format": "npz", "key": "points", "dtype": "float32", "shape": [52613, 4], "data": "..."},
 "filter_keep": {"format": "npz", "key": "filter_keep", "dtype": "bool", "shape": [52613], "data": "..."},
 "input_index": {"format": "npz", "key": "input_index", "dtype": "int32", "shape": [52613], "data": "..."},
 "class_names": ["drivable_surface", "other_flat_surface", "sidewalk", "manmade", "vegetation", "car", "..."],
 "class_counts": {"drivable_surface": 2927, "other_flat_surface": 20082, "sidewalk": 6, "manmade": 22903, "vegetation": 6546, "car": 13, "semi_trailer": 136, "...": 0},
 "meta": {"variant": "ot128", "num_points_input": 53411, "cropped_by_ego_box": 798, "num_points_interpolated": 0, "num_unique_coors": 52613,
          "filter": {"classes": ["drivable_surface"], "class_probability_threshold": 0.05, "kept": 44690}, "...": "..."},
 "timing_ms": {"decode": 25.078, "preprocess": 16.734, "device": 67.801, "postprocess": 3.418, "model_call": 87.992, "total": 117.089}}

• Points are in the sensor frame. Output order is the node's: the input points that survive the ego crop box, in input order (input_index maps them back to the request rows), then the points Autoware's interpolation appends (num_points minus num_points_raw; the QT128 network adds them).
• labels is Autoware's class_id (index into class_names), scores the softmax probability of that class, filter_keep the membership of the node's filtered cloud (~/output/pointcloud/filtered).
• A sweep outside the profile's ranges (ot128: 5,000-160,000 points, qt128: 5,000-120,000, both at most 60,000 cells) returns 200 with num_points: 0 and the node's message in meta.skipped, as the node publishes nothing.
• A Python API with the same outputs (tt_frnet.FRNet) is described in code/PYTHON.md; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Per-point classes computed by this package (ot128 profile) on 50 sweeps of PandaSet sequence 090 (El Camino Real, suburban, daylight, 10 Hz, a Pandar64 sweep inside the OT128's vertical field of view), with the front camera as an inset, also as MP4. The video was rendered with this port's initial version (same weights). Rendered from PandaSet, CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai); Scale AI and Hesai do not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (ONNX Runtime on the deployed ONNX, same Autoware pre- and post-processing), on every point the node scores:

The public frames are PandaSet sequences 019 and 029 (every 5th frame, Pandar64) and the 81 nuScenes v1.0-mini mini_val key-frames (HDL-32E), converted to the sensor frame; no public dataset has OT128 / QT128 sweeps or the 27 T4 labels, so these rows measure agreement, not accuracy. The weights repo publishes no dataset-level accuracy to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (50 warm iterations per stage) and 50 served requests of the sample to the package's server app run with uvicorn on the host; inside the container the server median over 350 requests of the sample was 116.8 / 96.6 ms. The model() and served times are host-bound: the host pre-processing takes 23.7 / 20.7 ms and request decoding 24.7 / 20.8 ms (ot128 / qt128).

Limitations

• Do not use it to drive or control a vehicle or as any safety function: it is not a certified Autoware component and has no functional-safety qualification. FRNet is an opt-in Autoware node that the default perception stack does not launch.
• Do not use its outputs as an FRNet accuracy benchmark or on other LiDARs without checking: the weights are TIER IV's fine-tune on about 16,000 frames of the non-public T4 dataset (Hesai OT128 / QT128), and accuracy on other sensors and domains is not characterized.
• Not a ROS 2 node: it serves HTTP and Python only. Inputs are arrays or files, not PointCloud2 messages, and the filtered cloud comes back as a mask (filter_keep).
• Batch 1, one sweep per request; requests are served one at a time; one network per process (the serve profile). A sweep runs padded to the smallest of three fixed capacities that holds it (ot128 40,000 / 80,000 / 160,000 points, qt128 29,984 / 60,000 / 120,000; at most 60,000 cells).
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Outputs were compared with the fp32 ONNX Runtime reference, not with the node's TensorRT engine (fp16 by default).
• Points near a class boundary can take the other class: the device's probabilities differ slightly from the fp32 reference, and on the out-of-domain nuScenes scene-0916 one frame per profile falls just below 98.5 % class agreement (98.494 % / 98.495 %).
• Autoware's CUDA races are replaced by fixed policies, so the output is deterministic: interpolated points are appended in row-major pixel order, and a duplicated cell keeps the last point (as ONNX Runtime does).
• Class names 22 / 23 follow Autoware's class_names (debris / stroller); the weights were trained with stroller = 22 and debris = 23.
• On public sensors the weights themselves (on CPU and on p150 alike) show the domain gap: with dataset labels mapped to the T4 classes, the fp32 reference reaches a coarse mIoU of 0.715 on PandaSet (Pandar64) and 0.483 on nuScenes (HDL-32E), the QT128 network 0.19 on PandaSet; traffic cones are almost never found and pedestrians next to structures often become manmade.

Licensing

• Weights: AutowareFoundation/lidar_frnet at tag v2.0 (commit a53a1c11b0b28d31fd50a03abef5db0e830a5b3c), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. Trained by TIER IV with AWML on the T4 dataset (about 16,000 frames); the upstream card states no further data licence notice.
• Pre- and post-processing ported from autoware_universe perception/autoware_lidar_frnet (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Samples code/tt_frnet/samples/synthetic_ot128.npz and synthetic_qt128.npz: synthetic scans generated by this package (code/scripts/make_sample.py, no dataset involved), Apache-2.0.
• Demo media in media/ are renders, not part of the package. The demo video and the other PandaSet renders: CC BY 4.0 and the PandaSet Dataset Terms (Scale AI and Hesai). The nuScenes renders media/*_NC.png: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc. Neither dataset owner endorses this work. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/frnet-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: