MoE-200M (Qwen3) — step 70 000 / 100B-run intermediate checkpoint
Intermediate checkpoint from the in-flight moe-200m-qwen3-100b pretrain run, captured at step 70 000 ≈ 2.4 B tokens consumed (2.4 % of the 100B-token budget). Trained autonomously by the ml-intern Claude Code skill on 2× Tesla V100-SXM2 32 GB.
This snapshot was extracted to power a browser-side WebGPU demo: → AlexWortega/moe-200m-qwen3-webgpu-demo
The model is not yet converged — the 100B run continues. The full final ckpt will land at AlexWortega/moe-200m-qwen3-100b-* once the run completes.
Architecture
DeepSeekMoE-style MoE with a Qwen3 tokenizer:
| total params | ~620 M |
| active params | ~200 M / token |
| vocab | 151 936 (Qwen3) |
| d_model | 640 |
| n_layers | 16 (layer 0 dense, layers 1–15 MoE) |
| attention | GQA — 10 Q heads / 2 KV heads, head_dim 64, partial RoPE (32 dims) |
| experts | 16 routed + 1 shared, top-2 sigmoid router |
| d_ff | 1024 (per expert) |
| tied embed/lm_head | yes |
| µP base_d | 512 |
| precision (training) | fp16 AMP, Muon + AdamW, WSD schedule |
The MoE dispatch uses a token-permuted, capacity-padded grouped-bmm kernel (moe_backend="grouped") — stacked expert weights of shape [E, d_ff, d] (gate, up) and [E, d, d_ff] (down). State-dict keys are flat blocks.{i}.ffn.{gate,up,down} rather than per-expert ModuleList entries; the legacy per-expert layout is also accepted (auto-stacked) by MoEModel.load_state_dict.
The router is a sigmoid + top-k pick with aux_coef=1e-3, z_coef=1e-3, and an additive bias controller (symmetric, with starved-expert boost, clamped to ±10). See model.py for the full SigmoidRouter + _moe_dispatch_grouped implementation.
Training state @ step 70 000
| tokens_seen | 2.29 B (≈ 2.3 % of 100 B target) |
| step | 70 000 |
| train lm_loss | ≈ 3.55 |
| eval loss | ≈ 3.48 – 3.55 |
| router CV | ≈ 0.79 |
| router entropy | 3.43 bits |
| dead expert slots | 21 / 240 (89 % capacity utilization) |
| throughput | ≈ 27 k tokens/s on 2× V100 |
| hardware | 2× Tesla V100-SXM2 32 GB (GPUs 2, 3) |
How to load
from huggingface_hub import snapshot_download
import importlib.util, json, torch
from safetensors.torch import load_file
local = snapshot_download(repo_id="AlexWortega/__REPO__") # replaced by hf_push
spec = importlib.util.spec_from_file_location("_m", f"{local}/model.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
cfg = mod.MoEModelConfig(**{k: v for k, v in json.load(open(f"{local}/config.json")).items() if not k.startswith("_")})
m = mod.MoEModel(cfg).eval()
sd = load_file(f"{local}/model.safetensors")
m.load_state_dict(sd, strict=False)
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(local)
ids = tok("Once upon a time,", return_tensors="pt").input_ids
with torch.no_grad():
logits, _, _ = m(ids)
print(logits.shape)
A self-contained load_test.py is included; run python load_test.py after pip install transformers safetensors huggingface_hub torch.
Caveats
- Mid-training snapshot. Loss is still descending at this step. Generation will be coherent at a sub-word / phrase level but not yet at long-range narrative level.
- The published
MoEModel.forward()returns(logits, loss, aux)when called without labels; the WebGPU export uses a simplified wrapper that only computes logits and never materializes aux losses. - ONNX export materializes all 16 expert outputs per token (path A: sparse-as-dense). This is wasteful at inference but standard ONNX-exportable. See the demo Space repo for the export code.
Reproducibility
The full training source (model.py, training scripts, train.log, eval.log, RESULTS.md) is available at the run's GitHub mirror: AlexWortega/moe-200m-qwen3-100b-.