This repository contains no model weights. It is a deployment recipe and a measurement report for running RadixArk/Qwen3.8-Flash-Next-NVFP4 across two DGX Spark under vLLM on Kubernetes. Mirror of the GitHub repo.

Qwen3.8-Flash-Next NVFP4 on 2× DGX Spark — vLLM on Kubernetes

Deployment of RadixArk/Qwen3.8-Flash-Next-NVFP4 across 2× NVIDIA DGX Spark (GB10, sm_121) with TP=2 + expert parallel, MTP-3 speculative decoding and CUDA graphs — running as Kubernetes Deployments, not docker run.

This is a companion to getrefined/Qwen3.8-Flash-Next-NVFP4-vLLM-DGX-Spark, whose launcher and PLE patch this deployment is built on. Same image, same checkpoint, same digest (vllm/vllm-openai@sha256:fc120ece…).

What this repo adds is the k8s translation and the things that only bite there — plus a list of levers that turned out not to work, so nobody else spends a night on them.

---

The one thing that will cost you a night: --headless

In vLLM multi-node with --distributed-executor-backend mp, the follower (--node-rank != 0) must be launched with --headless. Without it the follower starts a full server — API server and its own EngineCore — and that EngineCore calls collective_rpc("get_kv_cache_spec"), which is leader-only:

AssertionError: collective_rpc should not be called on follower node
  vllm/v1/executor/multiproc_executor.py:385

The follower dies. The head then hangs in an NCCL collective waiting for a rank-1 from a dead process group: 34 minutes with not a single log line and no restart, because a lenient startupProbe (120 × 30 s) never fires.

The symptom points nowhere near the cause. Things it is not, all measured before we found it:

The upstream recipe never hits this because it launches the follower itself with a separate docker run. In k8s both Deployments run the same launch script, so they have to be told apart:

if [ "${VLLM_NODE_RANK}" != "0" ]; then
  ARGS+=( --headless )
fi

--headless sets api_server_count=0 (serve.py:65-72); run_headless then sees node_rank_within_dp > 0 and builds the follower MultiprocExecutor — the branch the code itself comments as "Run headless workers (for multi-node PP/TP)" (serve.py:213).

Second thing: restart both ranks together

A normal rollout moves head and worker independently. The worker survives in a stale process group and the head crash-loops against it. Recycle both:

kubectl scale deploy -n llm qwen38-flash-next-head   --replicas=0
kubectl scale deploy -n llm qwen38-flash-next-worker --replicas=0
# wait for zero pods
kubectl scale deploy -n llm qwen38-flash-next-worker --replicas=1
kubectl scale deploy -n llm qwen38-flash-next-head   --replicas=1

---

Benchmarks

Read the caveats before the numbers. Measured with streaming, temperature 0, decode rate = (tokens−1)/(end − first_token) — i.e. excluding TTFT. Median of 4–6 reps.

Thinking off / reasoning_effort: low — responses that actually finish

Thinking on (default) — what we measured first, and why it was wrong

Those code/csharp figures do not measure code generation. With thinking on and a 4000-token budget the model spends the budget reasoning and returns empty content:

thinking on :  total=4000  finish=length  reasoning=15700 chars  content=0
thinking off:  total=2881  finish=stop    reasoning=0            content=10801

Acceptance tells the same story: 2.639 → 3.569 on code once it actually writes code. The draft predicts real code far better than a reasoning stream.

The decomposition that makes this legible

tok/s = steps/s × accept_length. Steps/s is constant across content types (16.3–16.6); every difference between categories is MTP acceptance. Formulaic maths accepts 3.43 tokens per draft, C# with XML docs and generics accepts 2.43.

If you compare two deployments, compare steps/s — comparing tok/s across different prompts compares different workloads.

Caveats, in order of how much they should worry you

1. Measurement noise is large. Identical config, identical prompt, temperature 0, 8 reps: 37.9 43.9 46.3 40.7 42.0 42.6 42.7 49.6 → mean 43.2, 27% range. With 6 reps per category σ drops to ~1.2–1.6 (≈3%). A config change is only credible above ~5%. We attributed five improvements and one regression before measuring this; all were inside the noise.
2. These numbers come from a cluster that also serves live traffic. Two later measurement runs were contaminated by 2–5 concurrent production requests and showed step rates collapsing to 7 steps/s. Check vllm:num_requests_running before you measure, not after you are surprised.
3. We do not claim a comparison against the upstream recipe. Their probes complete in 545–1200 tokens; ours ran to 4000+ with thinking on. Different output lengths are different workloads.

---

Falsified levers — measured, not argued

Everything here was tried on this hardware and did not help. Listed so you can skip them.

Where the time actually goes

torch.profiler on a real 150-token generation (4378 ms of kernel time):

Top kernel is cutlass_80_wmma_tensorop_bf16_s161616gemm_bf16_16x16_128 — 35 000 calls, 43%. Those are SM80 (Ampere) WMMA kernels running on SM121, and that is not a defect: for the skinny decode shapes (M=1–4) they hit 98–100% of the memory roofline. Forcing cuBLASLt or classic cuBLAS both made it slower. You cannot beat memory bandwidth with a better kernel.

The NVFP4 experts do use the modern path — MainloopSm120ArrayTmaWarpSpecializedBlockScaled (14.7%). The BF16 kernels are the modules the checkpoint excludes from quantisation (quantization_config.ignore: mtp.*, *.self_attn.*, *.linear_attn.*, *.mlp.gate*, *.mlp.shared_expert.*, lm_head), and with MTP-3 the draft runs 3 of the 4 forwards per step.

---

VLLM_USE_DEEP_GEMM=0

Preventive, credit to vllm#54125. On sm_121 support_deep_gemm() accepts the whole 120 capability family, so the gate opens on GB10 — verified here: support_deep_gemm() = True. With a blockwise-FP8 layer in front it selects DeepGemmFp8BlockScaledMMKernel and takes a CUDA fault during the startup profile_run.

It does not affect this checkpoint (quant_algo NVFP4, weight schemes {(4 bits, None, None)}, no FP8_PB_WO; zero Selected …BlockScaledMMKernel lines, zero DeepGEMM kernels in the profiler trace out of 173). Set anyway because it costs nothing and the failure mode is Engine core initialization failed. Failed core proc(s): {} with the DeepGEMM frame buried — the same generic message that hid the --headless bug for 34 minutes.

Also worth knowing

• NCCL_IB_HCA needs a leading = for exact match. Without it NCCL matches by prefix, so rocep1s0f0 also matches rocep1s0f1. With expert parallel this stops being cosmetic — the all2all sprays traffic across every listed port.
• The RDMA link autoneg does not survive a reboot on this hardware; pin it (ethtool -s … autoneg off speed 200000 on both ends) or the head sits in Init behind a misleading NFS error.
• The upstream README's sync; echo 3 > /proc/sys/vm/drop_caches before load is worth honouring on unified memory.

Environment

Files

• qwen38-flash-next-nvfp4-vllm.yaml — the full manifest, sanitised. Replace HEAD_USER, WORKER_USER, 10.0.0.x, example.com and the node names.

Credit

The launcher, the PLE FP8 patch and the original benchmark methodology come from getrefined/Qwen3.8-Flash-Next-NVFP4-vLLM-DGX-Spark. The DeepGEMM gate issue is vllm#54125, reported by someone else.