Research checkpoints of a naively upcycled Mixture-of-Experts model, released to study whether identical experts can differentiate. The dense model JIRU-W/dense_e1_seq2048_gbs128_10b at step 15000 was copied into a 32-expert top-8 MoE and training continued no load-balancing loss. Trained to the end of the schedule (step 38,139). Small research model, not instruction-tuned.
upcycle_dense.py, upcycle_meta.json)norm_topk_prob=True, so at step 15000 the MoE computes exactly the dense function| data | JIRU-W/olmoe-mix-0924-10b, continuing the dense run's data order from step 15000 (np.random.default_rng(1234).permutation(4881809), 128 samples / step) |
| batch | 128 × 2048 = 262,144 tokens / step |
| optimizer | AdamW with fresh state, weight decay 0.1; LR continues the dense run's cosine schedule (peak 8e-4, 200 warmup, decay to 8e-5 at step 38,139) and is additionally re-warmed linearly from 0 over the first 200 steps after step 15000 |
| load-balancing loss | none |
| precision | bf16 weights, fp32 gradient accumulation, DeepSpeed ZeRO-2, 1 GPU, seed 1234 |
main: newest HF checkpoint, step 38139init: the upcycled model before any MoE training (step 15000)step<N>: HF checkpoints (every 10 steps right after upcycling, then every 100, then every 1000)ds_step38139: DeepSpeed ZeRO-2 checkpoint with optimizer state at step 38139 (resumable; DeepSpeed >= 0.19.6), plus run_config.json, train_log.jsonl and the DeepSpeed configStep numbers count from the start of the dense run. Eval CE = mean token cross-entropy on 256 held-out 2048-token sequences ("—" = not evaluated at that step).
| revision | tokens seen (incl. dense phase) | eval CE |
|---|---|---|
step29000 |
7.60B | 2.9067 |
step30000 |
7.86B | 2.8943 |
step31000 |
8.13B | 2.8849 |
step32000 |
8.39B | 2.8741 |
step33000 |
8.65B | 2.8657 |
step34000 |
8.91B | 2.8582 |
step35000 |
9.18B | 2.8513 |
step36000 |
9.44B | 2.8450 |
step37000 |
9.70B | 2.8410 |
step38000 |
9.96B | 2.8368 |
step38139 |
10.00B | 2.8366 |
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JIRU-W/upcycle_dense15000_e32top8_noaux_rewarm200"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision="step34000", trust_remote_code=True, dtype=torch.bfloat16)
out = model(**tok("Hello", return_tensors="pt"), output_router_logits=True) # per-layer router logits in out.router_logits
JIRU-W/dense_e1_seq2048_gbs128_10b