Qwen3-8B-2layer-MoE-8E4A: Qwen3-8B DSpark MoE draft, 8 experts x 1536, top-4, batch-wise load-balancing loss + selection bias, 5-epoch run
Naming: [base model]-[layers]-[Dense/MoE]-[E experts, A active] = Qwen3-8B target, 2 draft layers, MoE with 8 experts and 4 active per token.
A 2-layer DSpark draft model for Qwen/Qwen3-8B whose feed-forward layers are
mixture-of-experts, balanced by a batch-wise load-balancing loss (coefficient 0.005) and a per-expert selection bias
(update rate 0.001): the recipe of SpecForge PR #1003 with four active experts per token
(num_experts_per_tok: 4). Counterparts with the same recipe: top-1
qwen3-8b-dspark-moe-batch-wise-0.005-selection-bias-only-5epoch
and top-2 qwen3-8b-dspark-moe-8expert-2activate-batch-wise-0.005-selection-bias-only-5epoch.
Trained for 5 epochs in one run (cosine schedule over 53,822 optimizer steps). Status: finished. One folder per epoch, <k>ep/,
each a servable export (config.json, model.safetensors, Qwen3-8B tokenizer). Every export is checked served-vs-trained before upload:
the MoE FFN of each layer is run through SpecForge's training MoELayer and through SGLang's DFlashMoE
(main + sgl-project/sglang#47181, the block Qwen3MoeDSparkModel serves) on the same
inputs; the chosen expert sets must agree on 100 % of the tokens and the outputs must differ by < 0.5 % (relative L2), so serving reproduces training.
Architecture
- 2 decoder layers, hidden 4096, 8 attention heads / 2 KV heads (head dim 128), block size 16, causal inside a block
- Conditions on Qwen3-8B hidden states from layers 1, 9, 17, 25, 33 (DSpark projector), vanilla Markov head (rank 256), no confidence head
- Each layer's FFN: 8 routed SwiGLU experts x 1536, top-4, no shared expert, softmax router without top-k renormalization (each pick weighted by its own softmax probability)
- About 0.48B parameters, 0.33B active per token (frozen target embedding / LM head not included); FFN FLOPs per token 4x the dense-1/8 draft
- Selection bias: fp32 per-expert bias (
layers.N.mlp.gate.selection_bias), added to the router scores only for the top-4 selection, moved by 0.001 per optimizer step toward under-loaded experts (zero-mean update); serving must apply it in the top-k selection
Training
- Code: SpecForge branch
exp/qwen3-8b-dspark-moe-8expert-4activate-selection-biasof https://github.com/yushengsu-thu/SpecForge, PR #13:configs/qwen3-8b-moe-8x1536-top4-dspark.json(the training node runsdev-moe-draft@475c9b92, the same code as the top-1 and top-2 runs, plus that config file) - Training log: W&B run
Qwen3-8B-2layer-MoE-8E4A(projectqwen3-8b-dspark) - Data: 1,420,903 Open Perfect Blend prompts with responses regenerated by Qwen3-8B (temperature 0, thinking off), qwen3-instruct template, max length 8192
- 5 epochs = 53,822 optimizer steps of 132 sequences (batch 2 x accumulation 11 x 6 DDP ranks); AdamW lr 6e-4, cosine, 4% warmup, grad clip 1.0, seed / prompt_seed 42
- Loss: 0.1 CE + 0.9 L1, 512 anchors, loss_decay_gamma 7; batch-wise load-balancing loss 0.005, selection bias 0.001 per step
- One 8x H200 node (specforge-h200-11051852): 2 SGLang capture servers (GPUs 0-1) + 6 trainer ranks (GPUs 2-7)
Per-epoch training metrics (mean over the last 1,000 optimizer steps of the epoch)
| Epoch | End step | Accuracy | Cross-entropy | Walk accept length | Load max/mean | Checkpoint |
|---|---|---|---|---|---|---|
| 1 | 10,764 | 0.647 | 1.539 | 4.39 | - | 1ep/ |
| 2 | 21,528 | 0.683 | 1.344 | 4.78 | - | 2ep/ |
| 3 | 32,292 | 0.700 | 1.258 | 5.00 | - | 3ep/ |
| 4 | 43,056 | 0.710 | 1.221 | 5.12 | - | 4ep/ |
| 5 | 53,820 | 0.719 | 1.182 | 5.24 | - | 5ep/ |
Updated 2026-11-08 22:30Z.