Qwen3-8B-2layer-Dense-FFN3072: Qwen3-8B DSpark dense draft, 2 layers, FFN width 3072, 5-epoch run
Status: finished.
The dense counterpart of the 2-layer MoE drafts at equal active parameters: one SwiGLU MLP of width 3072 per layer, which is the FFN width the 8 experts x 1536 top-2 MoE (Qwen3-8B-2layer-MoE-8E2A) activates per token (0.258 B active parameters for both; this model has 0.258 B parameters in total, the MoE 0.485 B). Same code, data and 5-epoch recipe as dense-1/8 (FFN 1536), the top-1, top-2 and top-4 MoE runs.
Architecture
2 decoder layers (full attention, causal inside the draft block), residual hidden 4096, GQA 8 heads / 2 KV heads (head_dim 128),
one SwiGLU MLP of width 3072 per layer, DSpark projector over target layers 1/9/17/25/33, vanilla Markov head (rank 256),
block size 16, no confidence head, target embedding and lm_head. Config: configs/qwen3-8b-dense-3072-dspark.json
(SpecForge fork branch exp/qwen3-8b-dspark-dense-3072-2layer, code dev-moe-draft @ 475c9b92 plus upstream SpecForge #958, #961 and #966 cherry-picked: fused FP32 CE + L1 logits loss, DDP sync once per accumulation window; consumer dispatch left at round_robin).
Training
- Data: 1,420,903 Open Perfect Blend prompts with responses regenerated by Qwen3-8B (temperature 0, thinking off), qwen3-instruct template, max length 8192
- 5 epochs = 53,822 optimizer steps of 132 sequences (batch 2 x accumulation 11 x 6 DDP ranks); AdamW lr 6e-4, cosine, 4% warmup, grad clip 1.0, seed / prompt_seed 42
- Loss: 0.1 CE + 0.9 L1, 512 anchors, loss_decay_gamma 7
- One 8x H200 node (specforge-exp-branch): 2 SGLang capture servers (GPUs 0-1) + 6 trainer ranks (GPUs 2-7)
- Training log: W&B run
Qwen3-8B-2layer-Dense-FFN3072(projectqwen3-8b-dspark)
Per-epoch training metrics (mean over the last 1,000 optimizer steps of the epoch)
| Epoch | End step | Accuracy | Cross-entropy | Walk accept length | Checkpoint |
|---|---|---|---|---|---|
| 1 | 10,764 | 0.624 | 1.666 | 4.15 | 1ep/ |
| 2 | 21,528 | 0.656 | 1.481 | 4.47 | 2ep/ |
| 3 | 32,292 | 0.670 | 1.406 | 4.65 | 3ep/ |
| 4 | 43,056 | 0.677 | 1.379 | 4.72 | 4ep/ |
| 5 | 53,820 | 0.685 | 1.341 | 4.82 | 5ep/ |
Serving
Each <k>ep/ folder is directly servable by SGLang 0.5.18: --speculative-algorithm DSPARK --speculative-draft-model-path <folder>.