diffusion-planner-p150 on Tenstorrent p150

Derived from AutowareFoundation/diffusion_planner (tag v5.0), the Diffusion Planner network that TIER IV trained for Autoware's autoware_diffusion_planner trajectory planner, with unmodified weights. Diffusion Planner: Zheng et al., arXiv:2501.15564. See the original model card for training details.

Diffusion Planner v5.0, a 14.55M-parameter diffusion-transformer planner: six MLP-Mixer entity encoders and a 6-block transformer fusion encoder over 564 scene tokens, a 3-block DiT decoder with adaLN (self-attention over the ego and 320 agents, cross-attention to the scene) evaluated 11 times by a 10-step DPM-Solver++(2M) loop, and a turn-indicator head; the whole plan runs on the device. The 15 planner tensors of the Autoware node go in (ego and neighbour histories, lanes with speed limits, route lanes, polygons, line strings, goal pose, ego shape, turn-indicator history, in the ego frame). Out come an 8 s ego trajectory (80 poses at 10 Hz with velocity and acceleration), an 8 s predicted path for every valid neighbour and a turn-indicator command, with the pre- and post-processing of the autoware_diffusion_planner node (input normalization, agent ordering, velocity smoothing, force stop, turn-indicator decision).

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Ego trajectory planning and neighbour motion prediction for Autoware-style driving stacks: research, prototyping and offline evaluation of recorded or simulated scenes, with the outputs of one autoware_diffusion_planner node. The caller builds the planner tensors from its own ROS messages and Lanelet2 map.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150

Quickstart

tt serve changh95/diffusion-planner-p150

tt serve downloads the Docker image and the AutowareFoundation/diffusion_planner at 423efde67f5414734da43a7ad856c17ceb8b51aa weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/diffusion-planner-p150.

Send the sample scene shipped in this repo (a scripted scene on Autoware's Kashiwanoha map, 88 neighbours) to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/diffusion-planner-p150/resolve/main/code/tt_diffusion_planner/samples/kashiwanoha_dense.npz
printf '{"inputs": {"format": "npz", "data": "%s"}}' "$(base64 -w0 kashiwanoha_dense.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

code/tt_diffusion_planner/server/client.py in this repo (standard library only) builds the same request from any .npz of the 15 tensors, with per-request params.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one planning instant (the 15 planner tensors) as JSON and returns the ego plan, the neighbour predictions and the turn-indicator command. GET /health and GET /info report the server and device state (/info lists the 15 tensors with their shapes); GET /v1/models is a stub, not a chat API.

Response for the sample scene (the trajectory cut to 3 of 80 rows, the predicted-agents array elided):

{"model": "diffusion-planner-p150", "frame_id": "base_link", "num_poses": 80,
 "columns": ["x", "y", "yaw", "cos", "sin", "velocity", "acceleration"],
 "trajectory": [[0.3644, 0.01, -0.0169, 0.9964, -0.0169, 3.8033, -0.0392], [0.7626, -0.0021, -0.0412, 0.9968, -0.0412, 3.7994, -0.5496], [1.1517, -0.0281, -0.0674, 0.9942, -0.0672, 3.7444, -0.4683], "... 77 more rows"],
 "turn_indicator": {"command": 1, "command_name": "DISABLE", "keep_selected": true, "held": false, "logits": [-15.96, -5.12, -5.235, -0.764, 5.502], "probabilities": [1.7e-09, 8.4e-05, 7.5e-05, 0.0066, 0.9932]},
 "predicted_agents": {"format": "npz", "key": "predicted_agents", "dtype": "float32", "shape": [88, 80, 5], "data": "<base64 npz>"},
 "meta": {"predicted_agent_columns": ["x", "y", "yaw", "cos", "sin"], "force_stop": false,
          "valid_counts": {"ego": 1, "neighbor": 88, "static": 0, "lane": 123, "route": 17, "polygon": 0, "line_string": 60, "goal": 1, "ego_shape": 1, "turn": 1}},
 "timing_ms": {"preprocess": 2.4, "device": 22.8, "postprocess": 1.2, "total": 36.9, "decode": 8.4, "model_call": 26.7}}

• trajectory: 80 poses at 0.1-8.0 s in base_link (the ego frame of the inputs), metres, radians, m/s and m/s², post-processed like the node's ~/output/trajectory: velocity from consecutive poses, a forward moving average over velocity_smoothing_window poses, force stop (poses frozen once the smoothed speed falls below stopping_threshold), acceleration by finite difference. yaw is what tf2::getYaw reads from the node's quaternion; cos / sin are the raw network outputs.
• turn_indicator.command: 0 NO_COMMAND, 1 DISABLE, 2 ENABLE_LEFT, 3 ENABLE_RIGHT.
• predicted_agents: one 80-point path (x, y, yaw, cos, sin) in base_link per non-empty neighbour row, in input order (meta.predicted_agent_rows).
• A Python API with the same outputs (tt_diffusion_planner.DiffusionPlanner) is described in code/PYTHON.md, including what the caller keeps between plans; it needs a tt-metal build with the two patches in this repo's patches/ directory.

Example output

Plans computed by this package for 33 consecutive planning instants of nuScenes scene-0061 (Singapore: following a van, then a left turn; 2 Hz, shown at 7 frames/s), drawn from above with the lanes, the neighbours' predicted paths and the ego plan. Also as MP4. Non-commercial use only: rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use; Motional does not endorse this work. Attributions for every file in media/: media/ATTRIBUTION.md.

Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing, same initial solver state); gates: ego plan max ≤ 1.0 m and mean ≤ 0.3 m over the 8 s, turn command equal, neighbour paths' median per-agent max ≤ 1.5 m:

Gate set: the 2 shipped sample scenes, 5 scripted research scenes and 92 planning instants converted from nuScenes v1.0-mini (Singapore and Boston). Held-out scenes, never used to choose the device numerics: the 33 consecutive instants of nuScenes scene-0061, 4 full-capacity and noisy-initial-state scenes and 13 scenes at every agent-bucket boundary. The open-loop rows compare with one human driver, not a planning benchmark; the paper's nuPlan closed-loop benchmark was not run, and the weights repo publishes no dataset-level metric to compare with.

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: code/scripts/bench.py (100 warm iterations per stage; device time = median of 3 rounds of 50 back-to-back trace replays) and 50 served requests of the sample scene (after 5 warm-up requests) to this package's server app run on the host with the serve settings. The device time depends on the number of neighbours: each plan runs the trace of the smallest agent bucket that holds the scene, with outputs equal to the full-capacity trace. The host stages (pre-processing, tensor packing, transfers, post-processing) take about 7 ms of the model() call, and request decoding 8.6 ms of the served time.

Limitations

• Do not use it to drive or control a vehicle, in closed loop or as any safety function: it is not a certified Autoware component and has no functional-safety qualification.
• Do not use its outputs as a Diffusion Planner benchmark: the weights are TIER IV's, trained on synthetic and real driving data that is not published. On nuScenes, a domain they were not trained on, the plans are plausible but conservative (moving plans about 17 % shorter than the logged drive), on CPU and p150 alike.
• Not a ROS 2 node: it serves HTTP and Python only. Building the 15 planner tensors from ROS messages and the Lanelet2 map is the caller's. The API is stateless: the node's turn-indicator hold window, the agent and ego histories and the initial-solver-state prefix between plans are the caller's, and the node's guidance services (start, stop, centerline guidance) are not available.
• Batch 1, one plan per request; requests are served one at a time. The v5.0 capacities (320 neighbours, 140 lanes, 25 route lanes, 10 polygons, 60 line strings) and 10 solver steps are fixed; the map entities are always computed at full capacity.
• One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
• Differences from the deployed node: the output stays in base_link (the node transforms it to map with the ego pose), the turn command is decided without the node's 1 s hold window, and the delay input is accepted and ignored (the node's multi-step mode does not read it). Outputs were compared with the fp32 CPU reference, not with TensorRT.
• Device numerics (fp32 residual streams, LayerNorm, attention and solver state; split bf16 + fp32 matmul operands; bf16 weights in the encoder MLPs) move the plan slightly from the fp32 reference, by a few cm on most scenes and up to 0.35 m (gate set) or 0.51 m (a synthetic scene with 192 neighbours) at the worst; the effect differs from scene to scene. Neighbour paths are gated only by their median, so single agents can differ more: on the kashiwanoha_dense sample the per-agent max difference is 8.6 cm median, 0.23 m at the 90th percentile and 3.72 m for the worst of 88 agents.

Licensing

• Weights: AutowareFoundation/diffusion_planner at tag v5.0 (commit 423efde67f5414734da43a7ad856c17ceb8b51aa), Apache-2.0 per its model card. Downloaded by tt serve at that commit, not redistributed here. The upstream card states that TIER IV trained the models on its synthetic and real driving data; the dataset composition is not published.
• Pre- and post-processing ported from autoware_universe planning/autoware_diffusion_planner (Apache-2.0).
• Port and serving code (code/) and the tt-metal patches in patches/: Apache-2.0.
• Sample scenes code/tt_diffusion_planner/samples/: kashiwanoha_dense.npz is derived from the Apache-2.0 Lanelet2 map AutowareFoundation/map-carla-kashiwanoha (0.2.0) with a scripted ego, route and agents; straight_road.npz is generated by this repo's code. Both Apache-2.0.
• Demo media in media/ are renders, not part of the package. The demo video and media/*_NC.* come from nuScenes: non-commercial use only, CC BY-NC-SA 4.0, © Motional AD Inc.; Motional does not endorse this work. The renders of the sample scenes: Apache-2.0. Full attributions: media/ATTRIBUTION.md.

Related packages

• The Autoware collection: the other Autoware perception and planning models packaged for the p150.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/diffusion-planner-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: