MoE-100M @ ~21.59B tokens (Ultra-FineWeb, graceful-stop ckpt of a 100B run)

This is the final ckpt from a 100B-token continuation pretrain of the ml-intern-moe100m-ufweb-en-1B model, stopped gracefully at step 131778 (≈ 21.59 B tokens consumed total counting the 1B and 10B warm-start phases). The originally- planned 100B target was abandoned because the held-out eval loss had flatlined at 2.91 for the last ~3.3B of training tokens — the model became data-limited on the Ultra-FineWeb-en slice well before the planned end of the run.

• Step: 131778 (saved as step_131778.pt / final.pt)
• Tokens consumed (total): 21,590,507,520 (21.59B)
• Architecture: 99.35M-active MoE (12L, d=512, GQA 8/2, 8+1 experts top-2, partial RoPE, QK-Norm, SwiGLU, tied 48k vocab); see model.py.
• Source ckpt: ckpts_100b/final.pt (raw weights — EMA shadow was scheduled to activate at step 549,323 and never reached).
• Trained autonomously by the ml-intern Claude Code skill.

Reproducibility bundle (TASK / PLAN_100B / OPS_100B / PLAN_21B / train+eval logs) is in this repo. The companion eval write-up will land as EVAL_21B.md in the run directory.

Why this ckpt

Continuation of an exploratory 100B-token run that was supposed to drive the held-out loss curve into a regime where downstream zero-shot eval would strictly dominate the 10B ckpt. Instead the loss saturated by ~18B tokens. This 21.6B ckpt is the honest "we kept feeding it data, here is what it learned" snapshot — comparable to the 10B ckpt at step66035-10B but with 2× the optimizer steps and a final dataset-pass beyond the early short-cycle scheduler.

Caveats

• Trained on Ultra-FineWeb-en only — does NOT speak Russian / code / multilingual.
• 100M active params on ~21.6 B tokens — data-limited; downstream evals show only marginal movement vs the 10B ckpt (see EVAL_21B.md).
• Vocab clipped at 48,000 (gpt2 tokenizer with last 2,257 ids collapsed). Don't feed token ids ≥ 48,000.