MoE-200M-active / 616M-total @ 11.96B tokens — ~12.0B-token intermediate ckpt
Sparse-MoE LM with ~200M active / ~616M total parameters,
trained from-scratch on Ultra-FineWeb (English split), tokenizer
Qwen/Qwen3-0.6B-Base
(vocab 151,936).
- Step: 365000
- Tokens consumed: 11,960,320,000 (11.960B)
- Architecture: see
model.pyandconfig.json. 1 shared + 16 routed experts top-2, GQA(10/2)×16 layers, partial RoPE, QK-Norm, SwiGLU, tied embeddings. - Trained autonomously by the ml-intern Claude Code skill.
Run summary
| key | value |
|---|---|
| param_count_total | 616,471,632 |
| param_count_active | ~203,591,000 (sum(p.numel()) minus inactive routed experts) |
| dataset | openbmb/Ultra-FineWeb (split=en, content) |
| tokenizer | Qwen/Qwen3-0.6B-Base (vocab=151,936) |
| step | 365000 |
| tokens_consumed | 11,960,320,000 |
Caveats
- Trained on Ultra-FineWeb-en only — does not speak non-English / code well.
- Intermediate checkpoint at ~10B tokens (near-Chinchilla for 200M active,
2.3× over for total params). Downstream evals attached in the run repo
(
EVAL_*.md). - Predecessor 100M-active run plateaued around 10B tokens on this dataset. This 200M-active model is the scaling follow-up.