Qwen3-8B DSpark draft: 5-layer dense, FFN 49152 (= the four active experts of 8E4A), 5 epochs
The iso-active dense counterpart of the 5-layer MoE 8x12288 top-4 draft (Qwen3-8B-5layer-MoE-8E4A-v1): the same 5 layers (GQA 32 heads / 8 KV heads x 128, target layers 1/9/17/25/33, markov head rank 256, no confidence head) with the MoE replaced by one dense SwiGLU FFN of intermediate size 49152 = 4 x 12288, i.e. the same active FFN FLOPs per token as 8E4A with half of its FFN parameters (0.60 B instead of 1.21 B per layer; ~3.3 B draft parameters in total = the 8E4A draft's active parameters, vs ~6.4 B total). It is the 5-layer counterpart of the 2-layer dense FFN-6144 draft (dense width = active width of the 8x1536 top-4 MoE) and the top-4 sibling of Qwen3-8B-5layer-Dense-FFN24576 (the 8E2A active width).
Everything else is the 8E4A recipe: PerfectBlend regenerated by Qwen3-8B (temperature 0, non-thinking), qwen3-instruct template,
8,192-token context, 5-epoch cosine from 6e-4, warmup 4 %, grad clip 1.0, DSpark CE 0.1 / L1 0.9, 512 anchors, decay gamma 7, global batch
132 (10,764 steps per epoch, 53,822 steps), seed 42, same code (dspark-v2-moe-launch @ 26bd18b6) plus sgl-project/SpecForge#963 with use_liger_kernel: true
(Liger fused RMSNorm/SwiGLU for the draft, a speed-only kernel choice).
Training started 2026-11-14 (PST) on two 4x GB300 devboxes in one NVLink domain: one SGLang capture server + three hybrid-FSDP trainer ranks
per node (6 ranks x batch 2 x accumulation 11 = 132 samples per optimizer step, sharded inside a node and replicated across nodes), Mooncake
over TCP between the nodes.
Config + recipe: branch
exp/qwen3-8b-5layer-dense49152-dspark (PR #20).
Training curves: W&B run.
Epoch checkpoints appear as 1ep/ ... 5ep/ (HF export, served by SGLang's Qwen3DSparkModel like any dense DSpark draft).