Speculative-decoding draft for Qwen/Qwen3-8B trained with SpecForge (DSpark). Every FFN is a MoE with 8 experts as wide as the dense draft FFN (12288), one active per token, no shared expert: 6.41 B total / 1.13 B active parameters, 1x the dense draft FFN FLOPs. Attention 32 q / 8 KV heads x 128 with q/k norm; projector fc 20480->4096 over target layers 1/9/17/25/33; Markov head rank 256; block 16.
Training (4 epochs, 2026-11-18 11:55 → 11-19 18:24 PST): perfectblend regen (temperature 0, no-think) prompts, 8192 context, qwen3-instruct template with no system prompt, lr 6e-4 cosine over the 4 epochs with 4 % warm-up, 144 samples per optimizer step (12 ranks x 6 x 2), 9,867 steps per epoch (39,469 in all), Liger RMSNorm, four 4x GB300 nodes (one node = 4 sglang 0.5.18 capture servers, three nodes = 12 trainer ranks), 2.77 s/step.
Accept length (block 7, c = 1, five tasks, request-count-weighted average; dense = jiapingW's 5-layer FFN-12288 draft):
| Avg (all requests) | gsm8k | math500 | humaneval | mbpp | mt-bench | Avg (w/o gsm8k) | |
|---|---|---|---|---|---|---|---|
| 4ep (best) | 6.109 | 6.422 | 6.354 | 5.646 | 5.466 | 4.271 | 5.727 |
| 3ep | 6.098 | 6.422 | 6.333 | 5.627 | 5.419 | 4.262 | 5.702 |
| 2ep | 5.983 | 6.333 | 6.222 | 5.422 | 5.245 | 4.112 | 5.556 |
| 1ep | 5.747 | 6.135 | 5.976 | 5.103 | 4.906 | 3.845 | 5.274 |
| dense (jiapingW) | 6.114 | 6.451 | 6.354 | 5.634 | 5.423 | 4.184 | 5.702 |
| Code: https://github.com/yushengsu-thu/SpecForge/pull/22 (training code, unchanged) + the draft config | |||||||
configs/qwen3-8b-5layer-moe8x12288-8A1-dspark.json (PR #22's 8x1536 top-1 config with moe_intermediate_size: 12288). |
|||||||
Checkpoints: 1ep/, 2ep/, 3ep/, 4ep/ (config.json + model.safetensors + tokenizer files). |
Serving with SGLang needs the fork's dev-moe-serving package (SGLANG_EXTERNAL_MODEL_PACKAGE=specforge.serving.sglang_models,
architectures Qwen3MoeDSparkModel) with the export's gate.selection_bias tensors renamed to gate.expert_bias and
dflash_config.moe_router_enable_expert_bias: true.
1ep/model.safetensors 13 GB2ep/model.safetensors 13 GB3ep/model.safetensors 13 GB4ep/model.safetensors 13 GBQwen/Qwen3-8B