Qwen3.8-Flash-Next on three DGX Spark: the "2 + 1" recipe
How we serve Qwen3.8-Flash-Next on three NVIDIA DGX Spark (GB10, 128 GB unified memory each), on top of ursuciprian's vLLM fork, with four pieces of our own. This is what has run in our production since 14 October 2026.
This repository holds no weights. It is the recipe, the patches, the tools and every test report. The weights are ursuciprian's and local-inference-lab's, pinned by revision below.
The detailed documentation is in Spanish, in docs/es/. This page is the English summary.
Results
Decode throughput is total server generation in tok/s: 256-token answers, 20 s of warm-up and 60 s measured at each concurrency level. Prefill is TTFT with no prefix cache, measured with llama-benchy.
| Before (our vLLM, TP=3, NVIDIA FP8 hybrid) | Now: TP=2 (2 Sparks) | Now: TP=1 (1 Spark) | |
|---|---|---|---|
| Decode, 1 / 4 / 8 / 16 requests | 49 / — / — / — | 73 / 158 / 226 / 341 | 43 / 102 / 137 / — |
| Cold prefill TTFT, 2K / 16K / 64K | — | 0.85 / 5.69 / 24.9 s | 1.38 / 8.19 / 32.6 s |
| KV cache | 3.7 M tokens (FP8) | 1.37 M tokens (FP8) |
Both servers together: 470 tok/s with 16 + 8 concurrent requests.
We also ran the same fork in TP=3 on all three Sparks. It is faster for a single request (81 tok/s), but the 2 + 1 split wins under load. With 24 concurrent requests of 32K cold prompts and 1024-token answers:
| 2 + 1 | TP=3 | |
|---|---|---|
| Server generation | 110 tok/s | 70 tok/s |
| Prefill | 3,977 tok/s | 2,541 tok/s |
| TTFT p50 / p95 | 96 / 180 s | 270 / 288 s |
| Requests completed in 10 min | 49 | 26 |
| Hottest thermal zone of the three Sparks, end of the run | 85.9 °C | 93.3 °C |
One of the Sparks runs with its GPU clock capped at 2200 MHz for thermal reasons. All figures, with their source files, are in docs/es/05-resultados.md and informes/.
What we put on top of ursuciprian's recipe
- SSM state in BF16 instead of FP32. The hybrid cache block drops from 2864 tokens to ~1500, so prefix caching hits more often and more context fits.
- "Mixed" GDN: b12x decode with BF16 state, triton prefill. The b12x decode kernel already accepts BF16 state; what blocked it was vLLM's backend-selection rule.
patch_gdn_mix.pyrelaxes that rule at boot. It is anchored and fail-closed: if an anchor does not match, the server does not start. - A Spanish MTP draft vocabulary. The 131,072 token ids the MTP draft head may propose come from our own traffic (Spanish, code, tool calls) instead of
ids-v2. The list is invocab/and the tooling incodigo/draft-vocab/. - A refusal mod: a per-request abliteration dial (
cache_salt: "refusal:λ") and a refusal detector. At λ=0 it changes nothing (exact-identity test).
Also: priority scheduling, KV cache fixed in bytes, and the Kubernetes operation around it.
We also built and measured these, and they are not in production:
- B2: 24 edits to b12x's CuTe kernels so the whole b12x GDN, prefill included, runs with BF16 state. It works on GPU. A tuned BF16 run is in progress.
- RoCE all-reduce on a switchless ring: ursuciprian's all-reduce pairs NICs by index, which only works with a switch.
codigo/roce-anillo/routes by peer instead, for three Sparks wired in a triangle. - A "universal" checkpoint view with the per-layer embedding (PLE) in NVFP4, valid for TP=1, 2 and 3. It saves 7 GiB per Spark against the FP8 PLE.
- The port of these pieces to our own vLLM, which we dropped in favour of the fork. It includes a b12x 1.3.0 wheel with ursuciprian's MoE micro-kernel fix.
The full list, with what each piece is and how it ended, is in docs/es/03-lo-construido.md.
How to reproduce
Image:
ghcr.io/ursuciprian/spark-vllm-b12x, by digest.- TP=2:
sha256:1346035488a5c8a0f405ade8e74febf68c66b8df1e026b7e3a6fb83cb9f4ce13(v2.1.0). - TP=1:
sha256:58e86bffedd500f8180daebfa52d6a55f80d596428153d2e5bff5112bbfeaf24(v2.3.0).
- TP=2:
Weights:
ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSEat revision03f4a057.local-inference-lab/Qwen3.8-Flash-Next-NVFP4at revision7c4f1bc1, used only for the MXFP8 copy taken for batches of 41 tokens or more.
Launch: run
receta/ursu/serve-gdn-mix.shwith:QWEN_GDN_MIX=A;GDN_TP=2(or1);- the environment in
receta/ursu/env-2x.env(orenv-1x.env); VLLM_MTP_DRAFT_VOCABpointing tovocab/ids-es-K131072.txt.gz.
The script patches the image and then starts ursuciprian's own launcher.
Refusal mod (optional):
codigo/refusal-b12x/, installed by itsrun.shbefore the launch.
receta/ is the Kubernetes overlay we run in production. It depends on our base manifests, so treat it as a reference for how the pieces fit together.
Repository layout
| Path | Contents |
|---|---|
docs/es/ |
Documentation (Spanish): summary, differences, everything built, timeline, results, lessons |
informes/ |
Every test report and raw result, by phase, with an index |
receta/ |
Production Kubernetes overlay and launch scripts |
codigo/ |
All our code: GDN patch, refusal mod, draft vocabulary, checkpoint view converter, RoCE ring patch, b12x port, benchmark tools |
vocab/ |
Spanish draft vocabulary (131,072 ids) |
b12x/ |
The b12x 1.3.0 wheel used in the port (Apache-2.0, with its license) |
SHA256SUMS |
Checksums of every file |
Credits
- ursuciprian: the
spark-vllm-b12xvLLM fork, the NVFP4 GDN-MSE checkpoint, SparkRun and the recipe all of this starts from. - b12x (Luke Alonso, Apache-2.0): the CuTe kernels for GB10.
- local-inference-lab: the NVFP4 checkpoint the MXFP8 copy comes from.
- Qwen: the model.
License
Our code and documentation are Apache-2.0. Third-party pieces keep their own licenses: b12x is Apache-2.0, and the weights and images belong to their authors.