pi05-base-p300x2

This package runs the Pi-0.5 VLA model of Physical Intelligence on two Tenstorrent Blackhole p300c boards (a QuietBox 2: 4 chips).

• Every chip runs the multi-config pi0.5 megakernel of changh95/pi05-base-p150: VISION, PREFIX and EXPERT as persistent ttnn.generic_op programs on 110 cores, one Metal trace per prompt bucket.
• The presets are built from parameters at server start:
  • cameras: 1/2/3/4
  • prompt bucket: 32/64/128/224, plus the extra buckets c1 L192, c2 L160, c4 L96 (each gives the bit-identical actions of the next standard bucket, with one VLM band less)
  • action chunk H: 1~64 (action-row bucket 32 or 64)
  • flow-matching steps N: 1 to 10
• Three serve profiles (one image; every request of a server uses the same cameras / H / N):

• Latency, one robot (224-token bucket, 64 action rows, N = 10; host-to-host p50 of write_inputs + replay + readback, tests/perf/bench_pi05_sp.py, cool chips, the code of this revision, 2026-10-13, mean of 2 runs; the one-chip row runs the same programs on one chip of the mesh):

Over HTTP (this image, single-robot, 2 cameras, 200-token prompt in the 224 bucket, cool): client p50 41.4 / p99 42.4 ms, server time p50 39.9 ms.

• Thermal steady state. The QB2's fans are fixed and under sustained load the hottest chip's GDDR reaches its 89 °C limit; the firmware then clamps that chip's AI clock to 800 MHz (instead of 1350). The sequence-parallel PREFIX waits for its slowest chip every layer, so the placement keeps the EXPERT and the cameras off that chip and gives it a 1-row slice of the PREFIX (at 4 cameras only the EXPERT moves). Per call (trace replay), at thermal steady state (legacy -> thermal placement): 2 cameras 61.4 -> 42.1 ms (-31 %), 20.1 -> 15.5 J; 4 cameras 80.6 -> 64.8 ms (-20 %), 27.8 -> 24.0 J. The cool latency changes by < 0.5 ms (c2 +0.4 ms).
• Multi-robot, 4 robots (2 cameras, 224 tokens, H 50, closed loop, in-process): p50 53 ms cool; at thermal steady state (3 of 4 chips clamped to 800 MHz) p50 83.0 / p99 84.1 ms, 52 requests/s, 6.65 J per request. With the profile's SLO (p50 100 / p99 110 ms, clock guard 1.69) the server admits 4 robots and refuses the 5th with 503; over HTTP (this image, cool, 60 s) the 4 robots got 69.7 requests/s at client p50 56.4 / p99 60.4 ms, none over the SLO.
• Stability. This revision fixes the main cause of the chip hangs seen on this box under multi-robot load (the PREFIX's 4 concurrent K / V multicasts; now one at a time, bit-identical, no time cost): ~11x fewer hangs in the first 20 min of load. A residual ~0.7 hangs per hour of 4-chip load remains; the server's hang watchdog keeps serving on the healthy chips. See Stability on QuietBox 2.
• Lower energy. Against the previous kernels (89d0ceb): one chip at 2 cameras -5.1 % time and -4.8 % energy per call hot (after 5 min of load; one run each, not interleaved); 4 robots -5 to -6 % energy per request and +5 % throughput. Outputs bit-identical.

Weights: lerobot/pi05_base · Paper: arXiv:2504.16054 · Upstream code: Physical-Intelligence/openpi · Single chip: changh95/pi05-base-p150

Quickstart

tt-model pull  changh95/pi05-base-p300x2 --with-weights
tt-model serve changh95/pi05-base-p300x2                          # multi-robot (default): SLO p50 100 / p99 110 ms
tt-model serve changh95/pi05-base-p300x2 --profile single-robot   # all 4 chips for one robot, lowest latency
tt-model serve changh95/pi05-base-p300x2 --profile disaggregated  # 3 prefix chips + 1 expert chip

printf '{"images":["%s","%s"],"prompt":"pick up the cube","state":[0.1,-0.2,0.3,0,0,0,0.5,-0.5],"robot_id":"arm-1"}' \
  "$(base64 -w0 media/sample_base.png)" "$(base64 -w0 media/sample_wrist.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

• The API is the API of the p150 package: POST /predict with images (exactly PI05_NUM_IMAGES, base camera first), prompt or tokens, optional state, seed and prompt_bucket; new: robot_id (multi-robot).
• multi-robot: send a stable robot_id per robot. A robot is bound to one chip; beyond R_max a NEW robot gets 503 with Retry-After (an admitted robot is never refused; a robot's slot frees after 5 s without requests). Requests without robot_id are routed per request and not admission-controlled.
• The response has batched_as (requests whose expert ran in the same program) and prompt_bucket.
• GET /info shows the layout, the chips' roles, the compiled prompt buckets, the SP placement (single-robot) or the SLO limits and latency counters (multi-robot: megakernel.slo).
• SERVING.md lists every environment variable; PERF_MULTICHIP.md every measured point.

Single-robot mesh (single-robot)

code/models/experimental/pi0/tt/ttnn_pi05_sp.py (PI05SingleRobotMesh) is the single-chip model built on a 1x4 FABRIC_1D_RING mesh. One call is 4 mesh launches (each over all 4 chips; an idle chip runs a no-op):

1. VISION: camera j's SigLIP on its own chip.
2. Relay: each image row tile to the chip whose PREFIX slice owns it (a local copy, or over the fabric).
3. PREFIX: each chip runs a contiguous slice of the prefix rows through the 18 VLM layers over ALL keys; its K / V feeder cores push the slice's K / V rows of every layer into the other 3 chips' caches inside the program.
4. EXPERT on one chip (every chip holds complete caches); the actions are read from it.

Every buffer and weight is replicated at the same address on the 4 chips. Outputs are torch.equal to the 1-chip model for all 32 standard presets (tests/pcc/test_pcc_pi05_sp.py, 1000-call closed loops included).

Thermal-aware placement (tt/sp_thermal.py; static, chosen once at open; logged as SP_PLACEMENT and shown in /info):

• The hottest chip (by default the board-position-0 chip measured to clamp first, found by its sysfs asic id, not /dev order) gets the smallest PREFIX slice (1 row tile at 1-3 cameras; c4 keeps 10/10/10/10), no camera when avoidable and not the EXPERT.
• The EXPERT goes to the coolest chip (another board, not the hot chip's ring neighbour).
• PI05_SP_HOT=<asic id | PCI BDF>,... names the hot chips on another box (none: no hot chip); PI05_SP_PLACEMENT=legacy restores the pre-thermal layout. Every placement is exact.
• Slices are capped at 10 row tiles: larger in-kernel exchange slices (12-14 rows) hung the mesh in testing.

Measured with energy_profile --mode sp (L224, 64 action rows, 143 tokens, interleaved arms, 2 runs each, every run from < 50 °C; 13-min arms: 10 min of whole calls, then the steady windows, equal to the last heat windows):

Multi-robot (multi-robot)

Every chip runs the whole model (tt/ttnn_pi05_multichip.py, layout="dp"); the policy is tt/mr_sched.py.

• Cycle model. k closed-loop robots bound to one chip fall into lockstep: each one's latency is the chip's cycle C(k) = k x (VISION + PREFIX) + batched EXPERT(k) + host time.
• Calibration at start. Every chip runs bursts of k = 1, 2, ... requests to measure C(k) and its jitter, until the SLO is missed. The robots per chip k_max is the largest k with C(k) x clock guard + host time <= SLO p50 and the same + margin <= SLO p99; R_max = 4 x k_max. PI05_SLO_CLOCK_GUARD=1.69 (the profile's) budgets for chips clamped from 1350 to 800 MHz under sustained load, so the limit holds at thermal steady state; PI05_MR_GUARD gives per-chip guards instead (opt-in).
• Routing and admission. A new robot is bound to the chip with the fewest robots and stays there; beyond R_max it gets 503.
• Chip loop. Prefills are chained; one batched expert runs for the robots whose prefixes are done. A request's actions do not depend on its batch (H > 32 or PI05_EXPERT_PACK=0: bit-identical to the single request).
• With the profile's SLO, the start-up calibration on this box (cool) gave a guarded cycle + host time of 93.3-93.9 ms (p50) for one robot per chip and 174.9-177.8 ms for two (three starts of the packaged server, the last one with this revision; 95.5 / 174.9 ms in a host run from the worktree): k_max 1, R_max 4. A looser SLO admits more robots per chip (each chip then batches their experts); e.g. at an SLO of 300 ms, 12 robots fit at zero think time (PERF_MULTICHIP.md).

Disaggregated (disaggregated)

Chips 0-2 run VISION + PREFIX from a shared queue; chip 3 runs the batched expert on up to 4 requests whose K / V caches (7.8 MB, 1.3 ms through the host) arrived. On this model it does not beat multi-robot: VISION + PREFIX are ~70 % of a request and cannot be batched, so either the expert chip (H 50) or the prefix chips (H 10) are the bottleneck (PERF_MULTICHIP.md).

Current limits: requests from any client are accepted at any time and the expert chip batches whatever prefixes are ready (up to 4), but all requests must share the server's one fixed configuration (cameras, H, N), like the other profiles; no SLO or admission control (4 clients: p99 101 ms). Per-request heterogeneous configurations and LLM-style continuous batching of heterogeneous robots that join and leave at any time are future work.

Faster and lower-energy programs

All bit-identical to the code they replace (code/models/experimental/pi0/tt/megakernel/README.md):

• Earlier in this package: wide VLM gate|up, norms fused into their matmuls, one VISION program for 3-4 cameras, the expert's merge weights on the diagonal faces.
• New: the fp32 residual kept in its owners' L1 across the residual ops (VISION, c1 / c2 PREFIX, the SP slices; PI05_PE_DEFINES=PE_NO_XRES=1 is the DRAM-path A / B arm), the last VLM layer's QKV computes K / V only, the key mask only on the attention parts that hold language / pad keys, the VLM down projection reloads its fp32 partial once per 2 K blocks, and the extra prompt buckets.

Against 89d0ceb (one run each, back to back, not interleaved, from < 50 °C; L224, 64 rows, N = 10; "hot (5 min)" = after 300 s of load, not the ≥ 12 min thermal steady state):

A busy chip draws ~125 W (it loses clock instead as it heats) and an open idle chip 60-80 W, so energy per call is close to 125 W x the device time; the savings are time.

Accuracy

Stability on QuietBox 2

Cause found and fixed. Under 4-robot load the previous release's kernels hung a chip within minutes on this box (fans fixed, tt-kmd 2.7.0, firmware 19.7.1). Captured while stuck: the NoC inside one chip stalled while the PREFIX's 4 K / V feeder cores multicast their quarters of each layer's K / V to the full grid at the same time; the 4 feeders waited in noc_async_write_multicast, the other ~96 cores waited for their flags. In this revision the feeders multicast one at a time (PE_KV_SERIAL), and each one starts only after every receiver acknowledged the previous feeder's data and flag (PE_KV_HANDOFF).

• Bit-identical. These tests are unchanged: the 1-chip golden of all 38 presets, the single-robot mesh exactness test (c1-c4) and the multi-chip vs single check.
• No time cost. Cool latency is 0.1-0.4 ms lower than with the concurrent feed (trace replay; c2 / c4: one chip -0.23 / -0.42 ms, mesh -0.14 / -0.21 ms). Multi-robot cool p50 is 53.1 vs 53.3 ms.

Hang rate measured after the fix. Hangs per hour of 4-chip load, Poisson 95 % CI. Setup: 4 robots, 2 cameras, 224 tokens; HTTP server soaks and probe runs, 2026-10-11 to 13.

• Early hangs: about 11x fewer (95 % CI 3-48x).
• Not fixed: a residual hazard remains.
  • 3 of the 4 remaining hangs were a chip dropping off the bus, which needs tt-smi -r. The 4th was the same NoC stall with a single multicast in flight.
  • The rate with hot chips has not been measured for this revision. An intermediate version (serial feed, no acknowledgement) had 2 hangs in 2.2 h there: 0.9 / h, CI 0.1-3.3.
• Single robot: a 27-minute soak through the server was clean (35705 requests, p50 45.8 / p99 53.6 ms).

The hang watchdog (server). Every profile sets these explicitly; the values are also the server defaults, except the marker path:

What a hang costs in multi-robot. Measured on real hangs: 4 robots, SLO 100 ms, degrade.

• Failed requests: only the 1 request in flight on the stuck chip fails. It gets 503 + Retry-After after ~10 s.
• The stuck chip: it is quarantined and its robot is re-bound to a healthy chip. /health reports degraded.
• The 3 healthy chips: they keep serving at p50 60-70 / p99 ~93 ms, ~41 requests/s (26 min in one run).
• The 4th robot: with 3 chips the SLO admits 3 robots, so it gets 503 + Retry-After until a restart. This is admission control, not a failed inference.
• Recovery: a quarantined chip does not come back in-process. The next start finds the marker, then resets and checks the chips (~51 s). The server is ready ~2.5 min after the restart. Measured in host runs; in this release's smoke, a container start with a marker on /cache reset and checked the 4 chips in 52 s and was ready after 166 s.

Recommended operation:

• Watchdog settings: keep PI05_HANG_S=10 and PI05_RESET_ON_START=auto.
• multi-robot: under a supervisor that restarts the server on exit code 75, set PI05_HANG_ACTION=exit; full 4-chip capacity is back in ~2.5 min. Without a supervisor, keep degrade, so that 3 chips keep serving.
• single-robot and disaggregated: run them under such a supervisor too. tt-model serve itself does not restart a container.
• Start timeout: give the supervisor one (~7 min; a normal start is ready in ~1-3 min). After a host reboot, 3 of 13 model builds on this box hung in the 1 GB weight upload. That is a host-to-device DMA stall, not these kernels, and the server has no watchdog on its build.
• Never run tt-smi, tt-triage or any other tool that opens the chips while the server runs.
  • Each such open sends the chips' firmware a power-setting message.
  • In 4 of 4 hang captures taken while the healthy chips were serving, those chips stopped within ~4 s of the tool opening the devices.
  • Read temperature and power from hwmon / sysfs (/sys/class/hwmon, /sys/class/tenstorrent) instead.
  • The server logs every other process that opens the chips.

Details: code/models/experimental/pi0/tt/megakernel/README.md ("Hangs"), and the watchdog in code/models/experimental/pi0_5/server/app.py.

Limitations

• One configuration per server (cameras, H, N fixed at start); every robot of a multi-robot server uses it.
• single-robot needs the 4 chips as a 1x4 ring with Ethernet links (it opens FABRIC_1D_RING itself) and the thermal placement is tuned on the box it was measured on: on another box, set PI05_SP_HOT to the chip that clamps first (energy_profile --mode sp warns when it is not the expected one).
• Thermal: the QB2 fans are fixed; expect the hottest chip(s) to clamp to 800 MHz after ~2-9 min of full load (the hottest chip after 2-3 min; the cool numbers above are the first minutes). Steady-state numbers are given where measured.
• The default multi-robot SLO (p50 100 / p99 110 ms, clock guard 1.69) fits one robot per chip at 1-2 cameras / H 50 on cool chips. At 3-4 cameras, or when started on hot chips (e.g. a restart right after sustained load), no robot fits and the server refuses to start. Raise PI05_SLO_MS / PI05_SLO_P99_MS, lower PI05_SLO_CLOCK_GUARD, or start from cool chips.
• Batch size: up to 6 / 4 requests per chip at 64 action rows (2 / 4 cameras), 8 at 32 rows (L1); the SLO sets the cap.
• Device hangs: a residual ~0.7 per hour of 4-chip load in the first 20 min (unmeasured later), handled by the watchdog (Stability on QuietBox 2); exchange slices > 10 rows also hang the mesh, and the code refuses them. Manual recovery: tt-smi -r (with nothing serving), then open / close the 4 chips.
• 224 x 224 images, 1-4 cameras, prompts of up to 224 tokens. The tokenizer google/paligemma-3b-pt-224 is gated: accept the Gemma terms and hf auth login, or send tokens.
• The outputs are normalized actions of the base checkpoint.

License

• Weights: lerobot/pi05_base, Gemma Terms of Use. Not included here.
• Code (code/): Apache-2.0 headers, distributed under the same Gemma terms.

Provenance