qwen3-32b-wh-galaxy

Qwen3-32B served on a Wormhole Galaxy through vLLM, packaged as a tt-model-manager container bundle.

• Model: Qwen/Qwen3-32B — 32.8B dense, 64 layers, GQA 64/8 heads with per-head QK-norm, Apache-2.0.
• Hardware: Wormhole Galaxy — 32 Wormhole chips in one 8×4 mesh (MESH_DEVICE=TG). tt-model labels this mesh n150x32; it is one Galaxy system, not 32 boards.
• Parallelism: 32-way tensor parallel prefill; decode as 4 data-parallel lanes × 8 users.
• Limits: 131,072-token context as configured (40,960 native), 32 concurrent sequences.
• Stack: tt-inference-server 0.21.0 release image (tt-metal bc4c4df, vLLM 0.25.1, vllm-tt-plugin) + the Wormhole Galaxy optimization set below, baked in with its serving defaults.
• Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1); listed in the community catalog (tt-model-catalog).

Quickstart

tt-model pull  changh95/qwen3-32b-wh-galaxy --with-weights
tt-model serve changh95/qwen3-32b-wh-galaxy

With tt-cli:

tt model pull changh95/qwen3-32b-wh-galaxy
tt serve changh95/qwen3-32b-wh-galaxy
tt model stop changh95/qwen3-32b-wh-galaxy

• pull --with-weights fetches the Docker image and the Qwen/Qwen3-32B weights at 9216db5781bf21249d130ec9da846c4624c16137 into your HF cache (weights are not in the image).
• serve starts an OpenAI-compatible server on port 20000 (or the next free port); ready when it logs Application startup complete.
• First start: ~25 min cold (kernel compile + trace capture for the whole mesh), ~8 min with a warm cache.
• Defaults: thinking off (enable_thinking: false), Qwen3 reasoning parser, Hermes tool-call parser, xgrammar structured outputs, seed 9472.
• Needs Docker ≥ 25, /dev/tenstorrent, and 1 GiB hugepages mounted at /dev/hugepages-1G.

What is in this package

• Prefix caching and async decode enabled for the Qwen class; prefix-affinity lane routing; cache-hit heuristics for the 4-column KV layout.
• Greedy requests on the batched 32-user prefill trace; traced 8192-token prefill; 256/512-token prefill buckets; shape-keyed row gather in the prefill output path (no per-length recompiles).
• Grouped prefix-cached prefill: up to four same-cached-length requests, one per mesh column, in one 4-row trace; collision-tolerant scheduler hold (1.5 s, four lanes, only above 24 active users).
• Cached-prefix attention with a 512-key chunk; bf8 sharded MLP weights; one 32768-wide top-k over both core rectangles in on-device sampling (below); one extra KV block per lane with a lane-aware block budget.
• Fused decode QK-norm kernel: one custom program per layer on the 16 head cores via ttnn generic_op, replacing a 14-op chain (~44 µs per layer); its constants are built in L1 per call and the norm weights arrive as runtime arguments, so the kernel reads nothing from DRAM (9.4 µs per layer inside the decode trace).
• Decode FF1/FF3 weights padded to a logical 3840 columns per device so the FF2-input all-gather takes the same fast collective as Llama-70B (12.6 → 7.9 µs per layer).
• Top-k reader kernel for on-device sampling (ttnn/cpp/.../topk/device/kernels/dataflow): the index tile is written as packed 32-bit pairs instead of 1024 sixteen-bit stores with read-backs, and the next input tile is read ahead. The two top-k ops of the sampling step drop from 689 + 370 µs to 212 + 132 µs (with the F36 split; the two ops are now one under the dual-grid factory below); the decode step loses about 0.6 ms at 32 users (source only, compiled by the JIT at first use).
• Boots under any host uid. tt-model serve runs the container as the invoking user; earlier bundles of this repo crashed at import unless that uid was the image's own (tt-metal's profiler helper creates a directory inside the source tree when ttnn is imported, which failed with EACCES and surfaced as a nanobind "type 'Layout' was already registered" abort). The directory and the home dot-directories are pre-created writable in this image; if you hit that abort on an older pull, tt-model pull changh95/qwen3-32b-wh-galaxy again.
• Prefill pipeline (round 4): per-length SDPA q-chunking (2048:64, 4096:32, 8192:64 tokens), the attention-output all-gather capped at 512 tokens, 56-core FF1/FF3 prefill matmuls, fused W1|W3 packing for short sequences (≤ 512 tokens, including the FF1/FF3 and attention matmuls), and the prefill tail (norm, sampling) reordered and traced. For the 2026-09-22 bundle, mean TTFT over the 54 grid cells was lower by 16 %, with no cell worse beyond the ±2 % / ±10 % (1 user) floor against the 2026-09-17 bundle.
• Engine and scheduler: the finished synchronous prefill is popped from the engine at once instead of waiting for the next step (a patched vLLM v1/engine/core.py, QWEN_POP_DONE_PREFILL=1); same-wave 128-token-class arrivals are joined within a 5 ms quiet window (QWEN_BURST_JOIN_MS=5); an arrival-rate hold of 8 µs per token of the longest waiting prompt (quiet window capped at 200 ms) that engages only when at least 2 requests are waiting, so a lone request shorter than 8,192 tokens is never held (QWEN_ARRIVAL_HOLD_US_PER_TOKEN=8, QWEN_ARRIVAL_HOLD_MIN_WAITING=2, QWEN_ARRIVAL_HOLD_MAX_MS=200); a lone prompt of 8,192 tokens or more and below 8,192 tokens is held 3 µs per token (24.6 ms at 8,192 tokens, capped at 200 ms) so the second request of a two-user wave joins its prefill call (QWEN_ARRIVAL_HOLD_LONE_MIN_TOKENS=8192, QWEN_ARRIVAL_HOLD_LONE_MAX_TOKENS=8192, QWEN_ARRIVAL_HOLD_LONE_US_PER_TOKEN=3); while a lane is already decoding, requests that arrived during a prefill call are gathered into one tail call as long as the newest arrival is younger than that quiet window or 40 ms, whichever is longer (QWEN_ARRIVAL_HOLD_TAIL=1, QWEN_ARRIVAL_HOLD_TAIL_MIN_MS=40); for waves whose longest waiting prompt is 16,384 tokens or more every hold releases as soon as 3 requests are waiting (QWEN_ARRIVAL_HOLD_LONG_TOKENS=16384, QWEN_ARRIVAL_HOLD_LONG_TARGET=3); at the KV-cache boundary of such a long wave on a busy mesh, when only some of the waiting requests would fit right now, admission waits (at most 3000 ms) until at least 3 of them fit, so a lone long straggler is not admitted mid-decode ahead of the rest of its wave (QWEN_ARRIVAL_HOLD_KV_BATCH=1). The hold shapes when a waiting prefill is admitted and never what the model computes.
• One 32768-wide top-k over both core rectangles in on-device sampling (QWEN_TOPK_SPLIT=0, QWEN_TOPK_GRID=dual): a rebuilt multi-rectangle top-k program factory (_ttnncpp.so and _ttnn.so rebuilt from four patched C++ sources under ttnn/cpp/ttnn/operations/reduction/topk/device, shipped with the two unchanged libtt_metal.so / libtt_stl.so of the same build as one set, md5 recorded in the manifest), a multi-rectangle final-reader kernel source and a sampler that reads the dual grid; about 0.17 ms per decode step at 32 users. The rebuilt libraries also carry an inert ring-matmul configuration that is off by default and measured flat.
• Prefill fold (round 5): same-length uncached prompts that arrive together are prefilled in one folded call (2/4/8/16 × 128 and 8/16/32 × 1,024 and 8/16 × 2,048 rows × prompt tokens) when the per-shape fold cost beats the serial estimate by at least 10 % (QWEN_PREFILL_FOLD=1, QWEN_PREFILL_FOLD_BUCKETS=1024:8,1024:16,1024:32,2048:8,2048:16,128:2,128:4,128:8,128:16, QWEN_PREFILL_FOLD_MIN_GAIN_PCT=10; QWEN_PREFILL_FOLD_COST=1024:8:577,1024:16:1094,1024:32:2155,2048:8:1043,2048:16:2043,2048:32:3975,2048:4:859,128:2:68,128:4:86,128:8:106,128:16:177 in ms per folded call, measured on the candidate lane on 2026-09-30, QWEN_FOLD_TAIL_MS=3.4 per-row sampling tail in the estimate); the 2/4/8/16 × 128 and 8/16 × 1,024 and 8 × 2,048 folds run as traces captured at start-up (QWEN_PREFILL_FOLD_TRACE_KEYS=1024:8,1024:16,2048:8,128:2,128:4,128:8,128:16); the 32 × 1,024 and 16 × 2,048 folds run eagerly; for the ≥ 2,048-token folds attention runs once per mesh column and is gathered back (QWEN_PREFILL_FOLD_COLATTN=1); up to two prefix-cached 128-token requests per mesh column share one grouped prefill call (QWEN_GROUPED_ROWS_PER_COL=2, trace 128_8_sp1); the device trace region is 450 MB (trace_region_size in tt-model.yaml). TTFT vs the 2026-10-01 bundle (f44), re-measured in this session, 0 % at 8 users × 1,024 tokens, +2 % at 16 users × 1,024 tokens, 0 % at 32 users × 1,024 tokens, -2 % at 8 users × 2,048 tokens, +1 % at 16 users × 2,048 tokens, 0 % at 2 users × 128 tokens, +1 % at 4 users × 128 tokens, 0 % at 8 users × 128 tokens, 0 % at 16 users × 128 tokens; time per output token in those cells unchanged within the 0.3 ms floor. Mean TTFT over the 54 grid cells vs the 2026-10-01 bundle (f44), re-measured in this session, is unchanged within noise (+0.6 %), with no cell worse beyond the ±2 % / ±10 % (1 user) floor widened by each cell's repeat spread against both that re-measurement and the 2026-10-01 card's published grid. Shipped as the .f45 overlay of generator.py: the previous bundle's generator plus three prefix-cache integrity fixes in the prefill scheduler (provider-before-consumer ordering, no rewrite of shared prefix blocks, warm-up guard); llama_model.py, llama_attention.py, llama_decoder.py, llama_mlp.py, worker.py and the fold planner qwen_fold.py are unchanged from the previous bundle. For the 2026-10-01 bundle (f44), TTFT vs the 2026-09-29 bundle (f43), re-measured in that session, was 0 % at 8 users × 1,024 tokens, 0 % at 16 users × 1,024 tokens, 0 % at 32 users × 1,024 tokens, -7 % at 8 users × 2,048 tokens, -13 % at 16 users × 2,048 tokens, 0 % at 2 users × 128 tokens, 0 % at 4 users × 128 tokens, 0 % at 8 users × 128 tokens, 0 % at 16 users × 128 tokens (time per output token in those cells unchanged within the 0.3 ms floor), and mean TTFT over the 54 grid cells was unchanged within noise (-0.2 %), with no cell worse beyond the ±2 % / ±10 % (1 user) floor widened by each cell's repeat spread against both that re-measurement and the 2026-09-29 card's published grid. For the 2026-09-29 bundle (f43), TTFT vs the 2026-09-25 bundle (f42), re-measured in that session, was 0 % at 8 users × 1,024 tokens, -1 % at 16 users × 1,024 tokens, 0 % at 32 users × 1,024 tokens, 0 % at 16 users × 2,048 tokens, -22 % at 2 users × 128 tokens, -46 % at 4 users × 128 tokens, -63 % at 8 users × 128 tokens, -45 % at 16 users × 128 tokens (time per output token in those cells unchanged within the 0.3 ms floor), and mean TTFT over the 54 grid cells was lower by 7 %, with no cell worse beyond the ±2 % / ±10 % (1 user) floor widened by each cell's repeat spread against both that re-measurement and the 2026-09-25 card's published grid. For the 2026-09-25 bundle (f42), TTFT vs the 2026-09-22 bundle (f40), re-measured in that session, was -13 % at 8 users × 1,024 tokens, -25 % at 16 users × 1,024 tokens, -31 % at 32 users × 1,024 tokens, -12 % at 16 users × 2,048 tokens (time per output token in those cells unchanged within the 0.3 ms floor), and mean TTFT over the 54 grid cells was lower by 1 %, with no cell worse beyond the ±2 % / ±10 % (1 user) floor widened by each cell's repeat spread.
• Prefix-cache integrity (round 5, f45): three fixes in the prefill scheduler close hazards around shared prefix KV blocks; all three are on by default and change nothing else. (1) Order guard: within one prefill step a request that reads another request's cached prefix is always run after the request that writes those blocks (a stable dependency sort over the step's work items; a cycle or any error falls back to one request at a time in scheduler order; QWEN_PREFILL_ORDER_GUARD=1). (2) No rewrite of shared prefix blocks: the KV fill tables skip every block that lies fully below a request's cached-prefix length, so a prefix block that already holds its provider's K/V is never written again by a folded, grouped or single-user prefill (QWEN_PREFILL_SKIP_SHARED_REWRITE=1). (3) Warm-up guard: the start-up prefill warm-up restores its state even when a warm-up step raises, so a failed trace capture can no longer leave the warm-up bypass on. Both knobs are code defaults of the .f45 generator (not serving-environment keys); with both off every served path is byte-identical to the previous bundle, and no fold bucket, cost table, trace key or serving knob changes.

How to make a query

OpenAI-compatible API on port 20000 (or the port tt-model serve --port chose); model id Qwen/Qwen3-32B.

tt-model curl "What is the capital of Australia?"      # one chat completion against the running model
curl -s http://localhost:20000/v1/models                # served model id

curl -s http://localhost:20000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in three sentences."}],
    "max_tokens": 256,
    "temperature": 0.6
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:20000/v1", api_key="none")
stream = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "Write a haiku about a lighthouse."}],
    max_tokens=128,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

• Thinking: off by default; per request, pass "chat_template_kwargs": {"enable_thinking": true} (Python: extra_body={"chat_template_kwargs": {"enable_thinking": True}}). Reasoning is returned in reasoning_content, the answer in content.
• Tools: pass tools=[...] and tool_choice="auto" (Hermes parser). Structured outputs: response_format with a JSON schema (xgrammar).
• Sampling: temperature, top-k, top-p and penalties run on the device. logprobs, per-token JSON constraints and min_p need host-side sampling and slow every user's decode while in flight — keep them off the hot path.
• Capacity: 32 concurrent requests (4 lanes × 8); more are queued. The first request after a restart may repeat its first token (known start-up quirk).

Performance

Measured on 2026-10-01 on a Wormhole Galaxy with this image's code, kernels, libraries and serving knobs (the serving lane bind-mounts the same files at the same paths):

• Through the OpenAI endpoint with vllm bench serve: random-token prompts, distinct per user; ignore_eos; one wave per point (as many prompts as users, all submitted together).
• Real weights, bf8 MLP weights and bf8 KV cache, greedy decoding, warm traces.
• Rows: concurrent users. Columns: input/output tokens per user.
• Per-user decode = 1000 / median over users of each request's time per output token ((end-to-end − TTFT) / (output tokens − 1)); aggregate = the wave's measured output-token throughput (1 user: 1000 / TPOT); TTFT = mean over users. The client's pre-run test request reuses the first prompt (a prefix-cache hit), so that request (index 0) is dropped from every statistic. Same definition as the 2026-10-01 card (f44); the previous-bundle comparison below re-measures that bundle's serving lane in this session with the same seeds and protocol instead of quoting the card's printed numbers; the 2026-09-17 card used the mean inter-token latency per request instead.

Per-user decode, tokens per second:

Aggregate decode throughput, tokens per second:

Time to first token, mean over users, ms:

• Users are prefilled one at a time, so mean TTFT grows with the user count; the 32-user 128-token points use the batched 32-user prefill trace, which is why they beat 16 users.
• Each lane's KV pool holds 131,648 tokens for its 8 users: at 16,384 tokens of context 7 users per lane are resident (28 of 32), at 32,768 only 4 (16 of 32). The 16- and 32-user cells in those columns are queued, not fully concurrent; their mean time per token is 15 to 60 % above the median shown.
• Previous bundle (2026-10-01 image, f44) in the same definition and with the same seeds, re-measured in this session, 54 shared cells: TTFT +0.6 % (equal-weight mean; 1 cells better and 0 worse beyond the ±2 % / ±10 % (1 user) floor widened by each cell's repeat spread against both that re-measurement and the 2026-10-01 card's published grid; worst u4 16384/128 +28.8 %, best u32 16384/128 -5.8 %), time per output token -0.01 ms (0 cells lower and 0 higher beyond the 0.3 ms floor widened by each cell's repeat spread, 1 more lower by 0.15-0.3 ms; worst u4 4096/128 +0.22 ms). 1 user 128/128: TTFT 61 → 61 ms, TPOT 12.94 → 13.02 ms; 32 users 128/128: TTFT 448 → 450 ms, TPOT 13.24 → 13.24 ms. No multi-user cell is worse against both the re-measured lane and the published grid. Per-cell tables with verdicts and cohort structures: perf_grid_f45_20261001_181847_vs_f44.md (vs the re-measured lane) and perf_grid_f45_20261001_181847_vs_f44pub.md (vs the published grid) (kept with the measurement data on the build host, not in this repository).

Closed-loop trace replay (a third party's protocol: ~8K-token prompts, 96 % sharing a cached prefix, 60 s warm-up + 240 s per point):

• This bundle: 60 / 220 / 384 / 624 / 1093 / 972 / 963 tokens per second at 1 / 4 / 8 / 16 / 32 / 64 / 128 users; 3879 requests, 0 failures.
• Previous bundle (2026-10-01 image, f44): 60 / 220 / 380 / 616 / 1087 / 958 / 954.
• Two bundles back (2026-09-29 image, f43): 60 / 219 / 386 / 622 / 1058 / 946 / 954.
• Three bundles back (2026-09-25 image, f42): 60 / 218 / 390 / 623 / 1051 / 953 / 960.
• Four bundles back (2026-09-22 image, f40): 58 / 218 / 382 / 623 / 1058 / 957 / 966.
• Five bundles back (2026-09-17 image, F36b): 60 / 217 / 382 / 614 / 1051 / 945 / 937.
• Six bundles back (2026-09-15 image, without the top-k reader kernel): 57 / 209 / 368 / 598 / 1032 / 926 / 925.
• Same image without the optimization set: 46 / 160 / 286 / 451 / 651 / 662 / 652.

Evaluation

Run on 2026-10-01 on the candidate serving lane of the previous bundle (f44: the same kernels, libraries, scheduler, serving knobs and model code except this bundle's generator.py overlay, bind-mounted at the same paths, with the prefill fold on) with tt-inference-server's --workflow evals (lm-evaluation-harness through the OpenAI completions endpoint; 3 of 3 tasks pass the workflow's accuracy check).

• Thinking mode on for every item (the R1-style task variants); Qwen3's recommended sampling: temperature 0.6, top-k 20, top-p 0.95, seeded; up to 32,768 generated tokens per item; 32 concurrent requests (16 for GPQA); 5 of 728 items (0 / 5 / 0) reached the token cap while still thinking and count as wrong.
• "± n" is the standard error of the mean over items. Published scores: Qwen's Qwen3 release blog for AIME 2024 and Artificial Analysis for MATH-500 and GPQA Diamond. The workflow grades each task against its configured reference score (80.0 for AIME 2024, the published score for MATH-500 and GPQA Diamond) and passes when score / reference is at least 1 minus the listed tolerance; AIME 2024 passes at 80.0 / 80.0 = 1.000 while sitting 1.7 % below the published 81.4.
• AIME 2024 is identical, item for item, to the 2026-10-01 build (f44 card, evals of 2026-10-01) (80.0 %; the same 30 items pass and fail); MATH-500 is identical, item for item, to the 2026-10-01 build (f44 card, evals of 2026-10-01) (94.2 %; the same 500 items pass and fail); GPQA Diamond is identical, item for item, to the 2026-10-01 build (f44 card, evals of 2026-10-01) (69.2 %; the same 198 items pass and fail).
• Measured on 2026-10-01 on the candidate serving lane of the previous bundle (f44), which bind-mounts this image's kernels, libraries, scheduler, serving knobs and model code except this bundle's generator.py overlay, with the prefill fold on; the three prefix-cache integrity fixes do not change what a single-user request computes, so the scores are re-used from that run rather than re-measured on this image.

Provenance

code/ in this repo is byte-identical to the model code inside the image.