Qwen3-8B DSpark draft: 5-layer MoE, 8 experts x 12288, top-2 (8E2A), selection-bias router (5 epochs)
Not started yet (queued on specforge-dev-branch after the 2-layer top-2 evaluation, 2026-11-06). The 5-layer 2x12288 draft
(configs/qwen3-8b-moe2x12288-2A1-dspark.json) with 8 routed SwiGLU experts of the full 12288 width and 2 active per token:
~6.4 B parameters, ~1.8 B active. Recipe: 5-epoch cosine from 6e-4, warmup 4 %, global batch 132, seed 42, batch-wise aux 0.005,
selection-bias update 0.001, fsdp_sharding: SHARD_GRAD_OP. Config + recipe: branch
exp/qwen3-8b-5layer-moe8x12288-8E2A-dspark.
Epoch checkpoints will appear as 1ep/ ... 5ep/ (HF export, served-vs-trained checked).