motif-3-bh-galaxy

Motif Technologies Motif-3 (314B MoE, 13.2B active, 384 experts + 1 shared, 1 MTP layer) served on one Tenstorrent Blackhole Galaxy (32 chips) through vLLM, with MTP self-speculative decoding.

Runs on p150x32 (mesh (4, 8), fabric FABRIC_2D_TORUS_XY): 32,768-token context, up to 32 concurrent sequences. Attention is 8-way tensor-parallel, the MoE 32-way expert-parallel.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5, self-contained bundle: no Docker needed).

Quickstart

export TT_MODEL_MODELS_DIR=/big/disk/tt-model   # ≈630 GB weights + ≈344 GB device weight cache
tt-model pull  changh95/motif-3-bh-galaxy --with-weights
tt-model serve changh95/motif-3-bh-galaxy

pull --with-weights installs the bundle and downloads the Motif-Technologies/Motif-3 weights at 2ed2ed5c (they are not in this repo). serve starts an OpenAI-compatible server on port 20000 (or the next free port); it is ready when it logs Application startup complete. The first start writes the device weight cache and compiles kernels (about 95 min); later starts take about 2 min.

Two launches, picked with MOTIF3_LAUNCH:

Variants: MOTIF3_LAUNCH=plain MOTIF3_ASYNC_DECODE=off tt-model serve changh95/motif-3-bh-galaxy (synchronous decode), and tt-model serve changh95/motif-3-bh-galaxy --no-enable-prefix-caching / MOTIF3_LAUNCH=plain tt-model serve changh95/motif-3-bh-galaxy --no-enable-prefix-caching (prefix caching off, as benchmarked).

Requirements: SFPI 7.84.0 (sfpi_install.sh in the bundle installs it without sudo), and tt-smi off your PATH when running tt-model (tt-model-manager up to 7994e0a does not know the board type tt-galaxy-bh).

Performance

tt-inference-server run.py --workflow benchmarks on one Blackhole Galaxy with real weights (BFP8 experts and KV cache), warm caches, prefix caching off, on the tt-metal commit this bundle was built from (2026-10-11). Greedy: random prompts, ignore_eos, temperature 0. Rows are concurrent users, columns are input/output tokens per user. A dash: not measured. Raw reports: reports/benchmarks/.

mtp launch (default), greedy. MTP acceptance 0.86-0.92.

Per-user decode, tokens per second:

Aggregate decode throughput, tokens per second:

Time to first token, mean over users, ms:

plain launch (asynchronous decode), greedy. TTFT ms / time per output token ms / aggregate tokens per second:

Sampled real text (TIS_BENCHMARK_VARIANT=sampled_text: SPEED-Bench prompts cut to 128 tokens, up to 1,024 output tokens, temperature 1.0, top-p 0.95). TTFT ms / time per output token ms / aggregate tokens per second:

Long prompts, 1 user (prefill is the same on both launches; measured 2026-10-08, prefill code unchanged since). Time to first token, s:

With prefix caching on (as shipped), 128/1024 time per output token: mtp 19.2 ms at 1 user and 35.6 ms at 32 users, plain 30.4 / 42.1 ms.

Evals

tt-inference-server run.py --workflow evals (lm-eval generation tasks, --limit-samples-mode ci-nightly subsets, plus repeats), sampled on the device at temperature 1.0, top-p 0.95, thinking on, outputs capped by the 32K context. Both launches at this bundle's code (2026-10-11). Raw reports: reports/evals/, analysis: reports/phaseG/evals/eval_analysis.json.

• All differences between the launches are within sampling noise (|z| ≤ 1.01); both pass the GPQA Diamond bar of 79.23.
• Speculative sampling is lossless: a distribution test of mtp with and without it passes at 1 user and with batch-invariant prefill at 32 users, and seeded requests give the same tokens with and without it (reports/phaseG/distribution_test/).
• plain with asynchronous decode (as shipped) gives the same tokens as synchronous decode (reports/two_launch/).
• Greedy, for reference (mtp speculating; the same scores as without MTP): MATH-500 96.0, AIME24 80.0, GPQA Diamond 77.5. Greedy scores are lower than sampled on this model, so serve with sampling.

Provenance

The exact sources the bundle was built from. metal/models in this repo is byte-identical to the model code of the tt-metal commit below.