Qwen3-8B-2layer-Dense-FFN6144: Qwen3-8B DSpark dense draft, 2 layers, FFN width 6144, 5-epoch run
Status: finished.
The dense counterpart of the 2-layer MoE drafts at equal active parameters: one SwiGLU MLP of width 6144 per layer, which is the FFN width the 8 experts x 1536 top-4 MoE (Qwen3-8B-2layer-MoE-8E4A) activates per token (0.334 B active parameters for both; this model has 0.334 B parameters in total, the MoE 0.485 B). Same code, data and 5-epoch recipe as dense-1/8 (FFN 1536), dense FFN-3072, the top-1, top-2 and top-4 MoE runs.
Architecture
2 decoder layers (full attention, causal inside the draft block), residual hidden 4096, GQA 8 heads / 2 KV heads (head_dim 128),
one SwiGLU MLP of width 6144 per layer, DSpark projector over target layers 1/9/17/25/33, vanilla Markov head (rank 256),
block size 16, no confidence head, target embedding and lm_head. Config: configs/qwen3-8b-dense-6144-dspark.json
(SpecForge fork branch exp/qwen3-8b-dspark-dense-6144-2layer, code dev-moe-draft @ 475c9b92 plus commit 84d0365 of that branch = upstream #961, the fused FP32 CE + teacher L1 logits loss, and #966, consumer dispatch kept on round_robin; both are speed-only changes).
Training
- Data: 1,420,903 Open Perfect Blend prompts with responses regenerated by Qwen3-8B (temperature 0, thinking off), qwen3-instruct template, max length 8192
- 5 epochs = 53,822 optimizer steps of 132 sequences (batch 2 x accumulation 11 x 6 DDP ranks); AdamW lr 6e-4, cosine, 4% warmup, grad clip 1.0, seed / prompt_seed 42
- Loss: 0.1 CE + 0.9 L1, 512 anchors, loss_decay_gamma 7
- 7 GPUs of an 8x B200 node (specforge-b200-11041712): 1 SGLang capture server (GPU 0, triton attention backend like the H200 runs, which used two such servers) + 6 trainer ranks (GPUs 2-7); GPU 1 of that box is thermally throttled and left unused
- Training log: W&B run
Qwen3-8B-2layer-Dense-FFN6144(projectqwen3-8b-dspark)
Per-epoch training metrics (mean over the last 1,000 optimizer steps of the epoch)
| Epoch | End step | Accuracy | Cross-entropy | Walk accept length | Checkpoint |
|---|---|---|---|---|---|
| 1 | 10,764 | 0.635 | 1.598 | 4.27 | 1ep/ |
| 2 | 21,528 | 0.671 | 1.400 | 4.65 | 2ep/ |
| 3 | 32,292 | 0.686 | 1.322 | 4.84 | 3ep/ |
| 4 | 43,056 | 0.695 | 1.285 | 4.95 | 4ep/ |
| 5 | 53,820 | 0.705 | 1.246 | 5.06 | 5ep/ |
Serving
Each <k>ep/ folder is directly servable by SGLang 0.5.18: --speculative-algorithm DSPARK --speculative-draft-model-path <folder>.