Qwen3-8B DSpark drafter, 5 layers, MoE 8 experts x 24576, top-4 (Setting 2, dim 24576)
Speculative-decoding draft for Qwen/Qwen3-8B trained with SpecForge (DSpark). Every FFN is a MoE with 8 experts of width 24576 (2x the dense draft FFN), four active per token, no shared expert: 12.45 B total / 6.41 B active parameters. Attention 32 q / 8 KV heads x 128 with q/k norm; projector fc 20480->4096 over target layers 1/9/17/25/33; Markov head rank 256; block 16. Dense counterpart: one FFN of width 98304.
Training (1 epoch): perfectblend regen (temperature 0, no-think) prompts, 8192 context, qwen3-instruct template with no system prompt,
lr 6e-4 cosine over the single epoch with 4 % warm-up, 144 samples per optimizer step (12 ranks x 3 x 4), 9,867 steps,
Liger RMSNorm, four 4x GB300 nodes. Code: https://github.com/yushengsu-thu/SpecForge/pull/22 (unchanged) + the draft config
configs/qwen3-8b-5layer-moe8x24576-8A4-dspark.json (the 8x12288 top-1 config with moe_intermediate_size / intermediate_size 24576 and num_experts_per_tok 4).
The checkpoint appears as 1ep/ when the epoch finishes. Serving needs the fork's dev-moe-serving package (Qwen3MoeDSparkModel,
gate.selection_bias renamed to gate.expert_bias, dflash_config.moe_router_enable_expert_bias: true).