Qwen3.8-27B-DSpine-b16-FP8

Tencent's DSpine drafter for Qwen3.8-27B with an FP8 backbone, plus an SGLang plugin that serves it next to the NVFP4 target nvidia/Qwen3.8-27B-NVFP4 on one DGX Spark (GB10, sm_121).

It is not a standalone model. It drafts 16-token blocks and the target verifies them. Greedy output is the target's argmax at every position, checked below.

Base: tencent/Qwen3.8-27B-DSpine-b16 (Apache-2.0). Our changes are listed in NOTICE.

Verdict on DGX Spark

Against our tuned DFlash2 launcher, same target, every pair measured on one Spark. Median [range], N runs:

• Wins at one user: 38.3 vs 33.9 [33.3-34.5] tok/s end to end, +13% (N=1 vs 2). Per user 45.8 vs 36.9.
• Ties at 4 users: 103.3 vs 105.1 (N=1 vs 2).
• Loses at 8 to 32 users: 153.9 vs 164.3 at 8 (-6%), 202.3 vs 240.3 at 16 (-16%), 259.1 vs 287.4 at 32 (-10%), N=3 to 4 each.

An earlier version of this card put the loss at -15% (8 users) and -16% (32 users), from one run per cell on Spark B, next to baseline ranges pooled across both Sparks. Those figures overstated the loss. Replicated runs on one Spark show -6% and -10%; a single 32-user cell moves 5% to 9% with acceptance. Every pair below is same-unit.

DSpine accepts more per round (3.7 to 3.8 vs 3.4 to 3.5 at 8 to 32 users) and still loses at load. Its draft round costs more than DFlash2's. Cheap structural trims of the drafter did not close the gap (see Trims).

What changed

• Weights. The five decoder layers and fc are FP8 (e4m3, one fp32 scale per output channel). The code bridge stays BF16. 2.32 GB instead of 4.05 GB. The FP8 values and scales were produced with SGLang's own online-quant kernel (per_token_group_quant_fp8, group = the row). The loader turns them into BF16 and --speculative-draft-model-quantization fp8 re-quantizes them bit-exactly, so serving equals SGLang's online FP8 of the original weights. Checked: same greedy tokens, logprobs and acceptance as the online path on 8 prompts.
• Draft head. Coarse candidates come from the target's own NVFP4 lm_head, as DFlash2's do. DSpine's 1.27 GB FP8 head copy is gone. The top-16 rescore and the transition tables use a BF16 copy of the target head, built from the target checkpoint at load.
• Verify widths. sps-gb10.json: Tencent's SPS tuner run on this stack for batch 1 to 7 (16, 16, 8, 8, 16, 16, 8), and 8 of the 16 drafted tokens at batch 8 and up (measured better at 32 users than the tuner's 16: 240.5 vs 216.6 tok/s on Spark B, one run each).
• Plugin. Ported from SGLang 0.5.22 to the SGLang nightly build c757c4b and to sm_121: FA4 draft attention (FA3 is Hopper only), bf16 GDN state, extra_buffer_lazy, up to 32 running requests, prefix-verify widths per batch, and a fix for NaN target probabilities in sampled verification on this GPU.

Results

Target nvidia/Qwen3.8-27B-NVFP4 (FP8 KV), 262,144 context, 32 running requests. Agent cells: 48 recorded agent turns, thinking on, temperature 1.0, top-p 0.95, top-k 20, 768 new tokens. tok/s is total output tokens over the cell, all users. Repeated cells start after one unmeasured 32-user pass, so each starts from the same warm prefix cache, with seeds 1234, 2234 and 3234. Baseline: our launcher with z-lab/Qwen3.8-27B-DFlash2, 8 draft tokens, online FP8 draft, window 2048. Both arms run SGLang nightly c757c4b with the same serving patches (prefill and scheduler changes that do not touch speculation; not part of this repo). The two arms ran interleaved.

Spark A, end-to-end tok/s, median [range] (N):

Spark A, acceptance (mean accepted tokens per round) and per-user tok/s, medians, same runs:

Spark B, one run per cell, earlier session:

Taking the coarse candidates from the target's NVFP4 head instead of DSpine's own 1.27 GB FP8 head copy moved 8 users from 122.6 to 151.5 tok/s on Spark B (one run each).

Trims

Cuts to the drafter that need no retraining, tried to reduce the work per round under load. Spark A, end-to-end tok/s, one run unless N is given:

None beats DFlash2 at 8 and 32 users, so none ships. The 8-row draft changes little because the shipped widths already verify 8 at 8 users and up. The shorter window and the thinner code bridge lose acceptance (window 2,048: 3.77 vs 4.29 at one user; last-layer bridge: 2.95 at 32 users). FP8 draft KV gains 3% at 8 users and loses 6% at 32. Cutting the decoder from 5 to 3 layers needs a distillation run and is not tried here.

Long context, single user, BF16 drafter on the same stack: 32K prompt first token in 15.0 s, 128K in 91 s. Prefill is the target's.

Exactness, this repo:

• Greedy: 0 of 1,580 emitted tokens differ from the target's own top-1 over 8 mixed prompts; two Sparks and the online-FP8 path gave token-identical output with zero logprob difference. Every trim above also held 0 of 1,543 to 1,580.
• Cache-resumed turns: 8 of 8 prompts identical cold and from the prefix cache, every token the target's top-1.
• Sampled: no degenerate replies in 192 sampled agent turns with the shipped bundle (384 with the online-FP8 runs).
• 262,144 context: a 260,013-token prompt prefilled in 269 s with both needles found (online-FP8 run, same numerics), free memory above 8.6 GiB.
• Free memory during the agent cells: 5.4 GiB at the lowest point of a 32-user cell with the shipped --mem-fraction-static 0.80, 8.9 GiB at 0.76 with its 32-user runs in the same range (244 to 263 tok/s). Use 0.76 if the box runs anything else.

Run it

docker build -f docker/Dockerfile -t sglang-dspine:c757c4b .
MODEL=/path/to/nvidia/Qwen3.8-27B-NVFP4 ./serve.sh

serve.sh mounts this repo as the drafter, loads sps-gb10.json, and applies sglang-patches/replay, the ReplaySSM commit fix this SGLang build needs for DFlash-family drafters with bf16 GDN state. Key flags: --speculative-algorithm DSPINE, --speculative-num-draft-tokens 16, --speculative-draft-attention-backend fa4, --speculative-draft-model-quantization fp8, --enable-linear-replayssm-spec, --mamba-ssm-dtype bfloat16, --mamba-radix-cache-strategy extra_buffer_lazy. The profile is tied to the GB10; retune with sglang-dspine/scripts/tune_sps.py on other hardware.

License

Apache-2.0, as the base. See LICENSE, NOTICE and sglang-dspine/THIRD_PARTY_NOTICES.txt.