MoE-200M (Qwen3) — step 160 000 / 100B-run intermediate checkpoint (5.24 B tokens)
Mid-training snapshot of the in-flight moe-200m-qwen3-100b pretrain run,
captured at step 159 999 ≈ 5.24 B tokens consumed (5.2 % of the 100 B-token
budget, +74 % more training than the step 70 000 / 2.4 B
snapshot).
Trained autonomously by the
ml-intern Claude Code skill
on 2× Tesla V100-SXM2 32 GB.
Source code, training log, full eval bundle: AlexWortega/moe-200m-qwen3-100b- (GitHub repo, in progress).
The model is not yet converged — the 100 B run continues. The final
checkpoint will land at AlexWortega/moe-200m-qwen3-100b-* once the run
completes.
Architecture
DeepSeekMoE-style MoE with a Qwen3 tokenizer:
| total params | 616.5 M |
| active params | 203.6 M / token |
| vocab | 151 936 (Qwen3) |
| d_model | 640 |
| n_layers | 16 (layer 0 dense, layers 1–15 MoE) |
| attention | GQA — 10 Q heads / 2 KV heads, head_dim 64, partial RoPE (32 dims) |
| experts | 16 routed + 1 shared, top-2 sigmoid router |
| d_ff | 1024 (per expert) |
| tied embed/lm_head | yes |
| µP base_d | 512 |
| precision (training) | fp16 AMP, Muon + AdamW, WSD schedule |
The MoE dispatch uses a token-permuted, capacity-padded grouped-bmm kernel
(moe_backend="grouped") — stacked expert weights of shape [E, d_ff, d]
(gate, up) and [E, d, d_ff] (down). State-dict keys are flat
blocks.{i}.ffn.{gate,up,down} rather than per-expert ModuleList entries; the
legacy per-expert layout is also accepted (auto-stacked) by
MoEModel.load_state_dict.
The router is a sigmoid + top-k pick with aux_coef=1e-3, z_coef=1e-3, and
an additive bias controller (symmetric, with starved-expert boost, clamped to
±10). See model.py for the full SigmoidRouter + _moe_dispatch_grouped
implementation.
Training state @ step 160 000
| tokens_seen | 5 242 880 000 (≈ 5.2 % of 100 B target) |
| step | 159 999 |
| train lm_loss (200-step window) | ≈ 3.47 |
| eval loss (held-out) | 3.458 |
| router CV | ≈ 0.64 |
| router entropy | ≈ 3.61 bits |
| throughput | ≈ 26.5 k tokens/s on 2× V100 |
| hardware | 2× Tesla V100-SXM2 32 GB (GPUs 2, 3) |
Zero-shot lm-evaluation-harness results
6 standard tasks, num_fewshot=0, batch_size=8, dtype=float16, Qwen3
tokenizer, single seed, no bootstrap. Δ is vs the step 90 000 / 3.0 B
snapshot of the same run.
| task | metric | random | gpt2-124M | pythia-160m | our-100M@21B | LFM2-350M | qwen2.5-0.5B | our-200M@3B | our-200M@5.2B (this) | Δ vs 3B |
|---|---|---|---|---|---|---|---|---|---|---|
| boolq | acc | 50.0 | 48.7 | 55.2 | 58.1 | 64.2 | 62.5 | 44.1 | 56.9 | +12.9 |
| hellaswag | acc | 25.0 | 28.9 | 28.4 | 31.7 | 38.4 | 40.6 | 28.7 | 29.7 | +1.0 |
| hellaswag | acc_norm | 25.0 | 31.2 | 30.3 | 35.9 | 49.0 | 52.2 | 31.1 | 32.2 | +1.1 |
| piqa | acc_norm | 50.0 | 62.5 | 61.4 | 64.2 | 69.5 | 69.9 | 57.6 | 57.1 | −0.5 |
| winogrande | acc | 50.0 | 52.4 | 51.0 | 50.0 | 55.7 | 56.5 | 49.4 | 50.2 | +0.8 |
| arc_easy | acc | 25.0 | 43.6 | 43.8 | 54.9 | 69.4 | 64.5 | 44.7 | 42.8 | −1.9 |
| arc_easy | acc_norm | 25.0 | 39.6 | 39.9 | 48.9 | 66.2 | 58.6 | 42.2 | 42.1 | ~0 |
| lambada_openai | acc | 0.0 | 32.2 | 32.7 | 23.5 | 40.2 | 52.5 | 17.8 | 16.6 | −1.2 |
| lambada_openai | ppl ↓ | — | 40.1 | 38.1 | overflow | 27.4 | 10.6 | 1064.3 | 984.6 | −7.5 % |
Average Δ over the 6 headline tasks vs the 3 B snapshot: +1.9 pt (3 of 6 positive). Boolq alone added +12.9 pt; piqa / arc_easy / lambada slipped by 0.5–1.9 pt (within single-seed noise band on these set sizes).
The headline qualitative event in this 2.24 B-extra-tokens window is boolq crossing chance (44 → 57) — eval_loss is still dropping monotonically (3.55 → 3.46) and lambada perplexity is now finite (was overflowing fp16 on our 100M @ 21B run), but most multi-choice tasks haven't moved much yet.
See EVAL_5B.md in the run-dir / GitHub repo for the full honesty pass.
How to load
A self-contained, runnable test ships with the repo:
pip install transformers safetensors huggingface_hub torch
python load_test.py
load_test.py does the full reload: snapshot-downloads this repo, imports
MoEModel from the bundled model.py, builds the config from config.json,
loads model.safetensors strictly, runs one forward pass, and samples 60
tokens from "Once upon a time,". A successful run prints LOAD_TEST: PASS.
Python:
from huggingface_hub import snapshot_download
import importlib.util, json, torch
from safetensors.torch import load_file
from pathlib import Path
local = Path(snapshot_download("AlexWortega/moe-200m-qwen3-step160000-5B-20260522-2256"))
spec = importlib.util.spec_from_file_location("_mdl", local / "model.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
cfg = json.loads((local / "config.json").read_text())
cfg = {k: v for k, v in cfg.items() if not k.startswith("_")}
config = mod.MoEModelConfig(**cfg)
config.router_noise_std = 0.0
config.use_liger_ce = False
config.use_chunked_ce = False
model = mod.MoEModel(config).to("cuda").to(torch.float32)
sd = load_file(local / "model.safetensors", device="cuda")
missing, unexpected = model.load_state_dict(sd, strict=False)
assert not unexpected
model.eval()
Reproducibility
model.py is the verbatim training-time file. config.json is generated from
the checkpoint's cfg dict (asdict on the MoEModelConfig dataclass) with
_model_class, _ckpt_step, and _tokens_seen metadata appended. The Qwen3
tokenizer is the unmodified Qwen/Qwen3-0.6B-Base tokenizer.
Caveats
- Mid-training snapshot — outputs are coherent at the sentence level but not yet competitive on multi-choice benchmarks vs reference 124–500 M models.
- Sigmoid router + grouped-bmm dispatch is the only supported backend at this scale; the legacy per-expert bmm backend would fall over on V100 throughput.
router_noise_std/use_liger_ce/use_chunked_ceare forced false inconfig.jsonfor inference; the training-time AMP / chunked-CE / Liger fused-CE paths are documented in the bundledmodel.pybut are not exercised byload_test.py.