Qwen3-32B-FP8 DFlash Draft Model
DFlash speculative decoding draft model for Qwen/Qwen3-32B-FP8. Trained using the DFlash (Block Diffusion for Flash Speculative Decoding) method from Z-Lab.
Architecture
| Parameter | Value |
|---|---|
| Draft layers | 5 |
| Hidden size | 5120 |
| Attention heads | 32 (8 KV heads, GQA) |
| Head dim | 128 |
| Intermediate size | 9728 |
| Block size | 16 |
| Target layers captured | [1, 16, 31, 46, 61] |
| Parameters (draft-only) | ~3.2B (bf16) |
| Tied embeddings | Yes (shared with target) |
The draft model takes concatenated hidden states from 5 target model layers as input and predicts a block of 16 tokens in parallel via iterative denoising. Attention is non-causal: queries attend to both target hidden states (context) and noise embeddings (draft tokens).
Training
| Detail | Value |
|---|---|
| Target model | Qwen/Qwen3-32B-FP8 |
| Dataset | ~50k multi-turn conversations (ShareGPT, OpenHermes, WildChat) |
| Max sequence length | 2048 |
| Effective batch size | 8 sequences/step (DDP across 8 GPUs) |
| Training steps | ~103k (1 epoch) |
| Hardware | 8x NVIDIA H200 141GB |
| Optimizer | AdamW, lr=4.8e-3, cosine schedule |
| Loss | Focal cross-entropy (gamma=7.0) |
| Precision | bf16 (draft), FP8 (target, frozen) |
Benchmarks
All benchmarks on a single NVIDIA RTX PRO 6000 Blackwell (98GB VRAM) using SGLang v0.5.13.post1.
ShareGPT (200 prompts, concurrency 8, max 1024 output tokens)
| Metric | DFlash | Vanilla | Speedup |
|---|---|---|---|
| Output throughput (tok/s) | 423.0 | 229.1 | 1.85x |
| Median TTFT (ms) | 88.0 | 313.2 | 3.56x |
| Median ITL (ms) | 14.7 | 34.6 | 2.35x |
| Median TPOT (ms) | 19.5 | 34.5 | 1.77x |
| Accept length | 2.46 | — | — |
Synthetic (random tokens, 100 prompts per config)
| Input/Output | Concurrency | DFlash tok/s | Vanilla tok/s | Speedup | Accept len |
|---|---|---|---|---|---|
| 128/128 | 1 | 54.5 | 20.7 | 2.64x | 2.15 |
| 128/128 | 8 | 299.0 | 205.4 | 1.46x | 2.13 |
| 128/128 | 32 | 578.3 | 605.8 | 0.96x | 2.13 |
| 512/512 | 1 | 71.0 | 20.9 | 3.40x | 2.47 |
| 512/512 | 8 | 438.6 | 215.5 | 2.04x | 2.58 |
| 512/512 | 32 | 814.3 | 631.2 | 1.29x | 2.63 |
| 1024/1024 | 1 | 77.8 | — | — | 2.77 |
| 1024/1024 | 8 | 478.3 | — | — | 2.83 |
| 1024/1024 | 32 | 869.0 | — | — | 2.87 |
| 2048/256 | 1 | 100.9 | — | — | 2.92 |
| 2048/256 | 8 | 555.0 | — | — | 2.96 |
| 2048/256 | 32 | 945.5 | — | — | 2.99 |
Accept length increases with context length (2.13 at 128 tokens to 2.99 at 2048 tokens).
Usage (SGLang)
python -m sglang.launch_server \
--model-path Qwen/Qwen3-32B-FP8 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path chutesai/Qwen3-32B-FP8-DFLASH \
--speculative-num-draft-tokens 16 \
--speculative-draft-attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.85 \
--host 0.0.0.0 --port 30000
Requires SGLang >= v0.5.13 with DFlash support.