A 3.7B-parameter (total) Mixture-of-Experts language model trained from scratch on 10B tokens with a per-layer
load-balancing loss. It is identical to JIRU-W/e32_top8_seq2048_gbs128_10b
(same code, initial weights, data, data order, seed and hyper-parameters; step-1 loss agrees to 1e-9) except for one switch:
the load-balancing loss is computed separately for each layer and averaged over layers (router_aux_loss_per_layer=True),
instead of the HF OLMoE/Mixtral form that pools router statistics of all layers by expert index. At uniform routing both
forms equal 8, so the coefficient (0.01) is unchanged. Training is in progress (step 20030 of 38,139 when this card was written); new checkpoints are added as they are saved. Small research model, not instruction-tuned.
OLMoE architecture (code copied from transformers v5.20.1 olmoe, class names renamed to MaskMoe*), shipped as custom code
(modeling_mask_moe.py, configuration_mask_moe.py); 16 layers, hidden 2048, 16 heads with QK-norm, RoPE θ=10000;
32 SwiGLU experts per layer (intermediate 1024), top-8, norm_topk_prob=False; vocab 50304 (OLMoE tokenizer);
3,696,887,808 parameters.
| data | JIRU-W/olmoe-mix-0924-10b: 10.0B tokens from allenai/OLMoE-mix-0924 (arXiv from EleutherAI/proof-pile-2), one pass, sample order np.random.default_rng(1234).permutation(4881809) |
| steps | 38,139 |
| batch | 128 × 2048 = 262,144 tokens / step |
| optimizer | AdamW, lr 6e-4, 200 warmup steps, cosine decay to 6e-5, weight decay 0.1 |
| load-balancing loss | coefficient 0.01, per layer (Switch loss per layer, averaged over the 16 layers) |
| precision | bf16 weights, fp32 gradient accumulation, DeepSpeed ZeRO-2, 1 GPU, seed 1234 |
main holds the newest checkpoint (step 20000); every checkpoint is also a branch step<N>
(steps 100, 200, 300, 500, 700, 1500, 2500, 3500, 4500 and every 1000). The final DeepSpeed checkpoint with optimizer state
will be added as branch ds_step38139 when training finishes. Eval CE = mean token cross-entropy on 256 held-out
2048-token sequences ("—" = not evaluated at that step).
| revision | tokens seen | eval CE |
|---|---|---|
step7000 |
1.84B | 3.1408 |
step8000 |
2.10B | 3.1094 |
step9000 |
2.36B | 3.0817 |
step10000 |
2.62B | 3.0595 |
step11000 |
2.88B | 3.0381 |
step12000 |
3.15B | 3.0184 |
step13000 |
3.41B | 3.0033 |
step14000 |
3.67B | 2.9856 |
step15000 |
3.93B | 2.9691 |
step16000 |
4.19B | 2.9552 |
step17000 |
4.46B | 2.9390 |
step18000 |
4.72B | 2.9256 |
step19000 |
4.98B | 2.9129 |
step20000 |
5.24B | 2.9005 |
Requires transformers>=5.20.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JIRU-W/e32_top8_seq2048_gbs128_10b_perlayeraux"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision="step20000", trust_remote_code=True, dtype=torch.bfloat16)