Qwen3-8B DSpark draft: 5-layer MoE, 8 experts x 12288, top-4 (8E4A), selection-bias router (5 epochs)
Training started 2026-11-08 17:07 PST on one 8x H200 node (2 SGLang capture servers + 6 FSDP trainer ranks, 10.9 s/step); from the
step-2,691 checkpoint (2026-11-09 01:52 PST) it continues on two 8x H200 nodes — 2 capture servers + 6 trainer ranks per node, 12 ranks
with hybrid FSDP sharding (sharded inside a node, replicated across nodes), 5.8–6.5 s/step — so the 53,822 optimizer steps of 132 samples
should finish around 2026-11-12 (PST) instead of 11/15. The restart kept the same samples-per-step, schedule and sample order (the producer skipped
the 2,691 x 132 prompts already trained), and the first two-node step matched the single-node run (acc 0.629 vs 0.624, lr identical). The 5-layer 8x12288 top-2 draft
(Qwen3-8B-5layer-MoE-8E2A-v1) with 4 experts active per token instead of 2:
8 routed SwiGLU experts of the full 12288 width, ~6.4 B parameters, ~3.3 B active (8E2A: ~1.8 B).
Everything else is the 8E2A recipe: 5-epoch cosine from 6e-4, warmup 4 %, global batch 132, seed 42, batch-wise aux 0.005,
selection-bias update 0.001, norm_topk_prob: false, no shared expert, fsdp_sharding: SHARD_GRAD_OP, same code (dspark-v2-moe-launch @ 26bd18b6).
One runtime difference: the 132-sample optimizer step is batch 1 per rank (8E2A: batch 2) because top-4 at batch 2 peaked at 138 GB per
trainer GPU in the 20-step smoke — 6 ranks x 22 accumulation steps on one node, 12 ranks x 11 on two nodes; the optimizer sees the same samples per step.
Config + recipe: branch
exp/qwen3-8b-5layer-moe8x12288-8E4A-dspark (PR #16).
Training curves: W&B run.
Epoch checkpoints appear as 1ep/ ... 5ep/ (HF export; each one passes the served-vs-trained check: SpecForge MoELayer vs SGLang DFlashMoE,
expert-set agreement 1.000 and relative |dy| ~4.5e-3 on the smoke export).
Evaluation (block 7 = gamma 7 / verify 8; SGLang main c10e0e17 + sgl-project/sglang#47181; one H200, TP=1)
Final checkpoint: ep4/ — the best pooled accept length among the epoch-3..5 exports (sweep below).
Protocol (the same as the other 5-layer MoE drafts): five tasks (gsm8k 1319, math500 500, humaneval 164, mbpp 257, mt-bench 160 prompts),
thinking off, temperature 0, max_new_tokens 2048, context 16384, radix cache off; accept length at concurrency 1, measured once;
throughput = mean ± sample std over 5 per-task + 5 mixed groups per draft and kernel config, one server per empty GPU. The dense
reference (jiapingW/Qwen3-8b-dspark-SpecForge: 5 layers, dense FFN 12288) was measured on the same box with the same stack.
Served-vs-trained check (SpecForge MoELayer vs SGLang DFlashMoE): PASS on every epoch export (top-4 expert-set agreement 1.000).
Accept length per epoch (c=1; pooled = request-weighted mean over the 2,400 requests, equal-weight = mean of the five task means)
| folder | optimizer step | epoch | gsm8k | math500 | humaneval | mbpp | mt-bench | pooled (all requests) | equal-weight (per task) |
|---|---|---|---|---|---|---|---|---|---|
| dense reference (jiapingW/Qwen3-8b-dspark-SpecForge, 5 layers, FFN 12288) | 6.451 | 6.354 | 5.634 | 5.423 | 4.184 | 6.114 | 5.609 | ||
1ep/ |
10764 | 1 | 6.197 | 6.073 | 5.235 | 4.973 | 3.877 | 5.820 | 5.271 |
2ep/ |
21528 | 2 | 6.398 | 6.319 | 5.598 | 5.341 | 4.199 | 6.067 | 5.571 |
3ep/ |
32292 | 3 | 6.488 | 6.465 | 5.759 | 5.560 | 4.341 | 6.191 | 5.722 |
4ep/ (final) |
43056 | 4 | 6.537 | 6.514 | 5.875 | 5.661 | 4.436 | 6.253 | 5.805 |
5ep/ |
53820 | 5 | 6.526 | 6.493 | 5.870 | 5.634 | 4.429 | 6.239 | 5.791 |
Accept length, ep4/ vs the dense draft
| Avg (all requests) | Avg (per task) | gsm8k | math500 | humaneval | mbpp | mt-bench | |
|---|---|---|---|---|---|---|---|
| dense | 6.114 | 5.609 | 6.451 | 6.354 | 5.634 | 5.423 | 4.184 |
| MoE | 6.253 | 5.805 | 6.537 | 6.514 | 5.875 | 5.661 | 4.436 |
| diff | +0.140 | +0.196 | +0.086 | +0.161 | +0.241 | +0.238 | +0.252 |
| diff % | +2.3% | +3.5% | +1.3% | +2.5% | +4.3% | +4.4% | +6.0% |
Mixed-stream throughput (the five tasks shuffled into one stream of 2,400 requests), output tokens / s
| concurrency | dense | MoE, default fused-MoE config | vs dense | MoE, sep-tuned configs (TMA) | vs dense | step ms dense / default / TMA |
|---|---|---|---|---|---|---|
| 1 | 730 ± 1 | 591 ± 1 | -19.0% | 600 ± 1 | -17.8% | 8.0 / 10.2 / 10.1 |
| 8 | 4986 ± 18 | 3448 ± 5 | -30.8% | 3965 ± 15 | -20.5% | 9.4 / 14.0 / 12.1 |
| 16 | 8425 ± 34 | 5755 ± 33 | -31.7% | 6750 ± 37 | -19.9% | 11.1 / 16.6 / 14.2 |
| 32 | 11320 ± 61 | 7948 ± 44 | -29.8% | 9360 ± 71 | -17.3% | 16.3 / 23.8 / 20.2 |
| 64 | 13579 ± 88 | 9690 ± 45 | -28.6% | 11522 ± 91 | -15.1% | 26.8 / 38.6 / 32.5 |
| 128 | 15245 ± 136 | 10714 ± 65 | -29.7% | 13146 ± 113 | -13.8% | 47.4 / 69.1 / 56.3 |
| 256 | 16387 ± 84 | 11512 ± 86 | -29.7% | 14177 ± 92 | -13.5% | 86.7 / 126.8 / 102.7 |
Fused-MoE Triton kernel configs for the draft's expert shape (E=8, N=12288, H200, triton 3.8.0), tuned on the draft's own top-4
routing (benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py --enable-tune-up-tma; USE_TMA for the up kernel at
M = 1, 2, 4, 8, 16, 24, 32, 48, 64, 96, 128, 256, 512, 1024, 1536, 2048, 3072, 4096, for the down kernel at M = 1, 2, 4, 8, 16, 24, 32, 48, 64, 96, 128, 256, 512, 1024, 1536, 2048, 3072, 4096): kernel-configs/E=8,N=12288,device_name=NVIDIA_H200.json and
kernel-configs/E=8,N=12288,device_name=NVIDIA_H200_down.json — copy them into SGLang's
python/sglang/srt/layers/moe/moe_runner/triton_utils/configs/triton_3_8_0/ to serve this draft with the tuned kernels.