dense_e1_seq2048_gbs128_10b
A 575M-parameter dense language model trained from scratch on 10B tokens, released with its intermediate checkpoints. It is the dense counterpart of the MoE model JIRU-W/e32_top8_seq2048_gbs128_10b: same architecture code, data and data order, but each layer has a single feed-forward block (one "expert") and no router. Its checkpoints are used as the starting point for upcycling experiments (copying the MLP into 32 experts). Small research model, not instruction-tuned.
Architecture
OLMoE architecture (code copied from transformers v5.20.1 olmoe, class names renamed to MaskMoe*), shipped as custom code.
With num_experts == 1 each layer's feed-forward is a plain SwiGLU MLP (gate_proj / up_proj / down_proj):
no router, no token dispatch, no load-balancing loss.
| layers | 16 |
| hidden size | 2048 |
| attention heads | 16 (no GQA), QK-norm, RoPE θ=10000 |
| feed-forward | SwiGLU, intermediate size 1024 |
| vocab | 50304 (OLMoE tokenizer allenai/OLMoE-1B-7B-0924) |
| parameters | 575,277,056 |
Training
| data | JIRU-W/olmoe-mix-0924-10b: 10.0B tokens from allenai/OLMoE-mix-0924 (arXiv from EleutherAI/proof-pile-2), DCLM 94.9% |
| data order | sample order np.random.default_rng(1234).permutation(4881809), identical to the MoE model |
| steps | 38,139 (one pass) |
| batch | 128 sequences × 2048 tokens = 262,144 tokens / step |
| optimizer | AdamW, lr 8e-4, 200 warmup steps, cosine decay to 8e-5, weight decay 0.1 |
| precision | bf16 weights, fp32 gradient accumulation, DeepSpeed ZeRO-2, 1 GPU |
| seed | 1234 |
Checkpoints
main holds the final checkpoint (step 38139). Every other checkpoint is a branch named step<N>.
Eval CE = mean token cross-entropy on 256 held-out 2048-token sequences (the eval split of the dataset).
| revision | tokens seen | eval CE |
|---|---|---|
step1000 |
0.26B | 3.8858 |
step2000 |
0.52B | 3.6322 |
step3000 |
0.79B | 3.5172 |
step4000 |
1.05B | 3.4438 |
step5000 |
1.31B | 3.3911 |
step6000 |
1.57B | 3.3552 |
step7000 |
1.84B | 3.3249 |
step8000 |
2.10B | 3.2972 |
step9000 |
2.36B | 3.2754 |
step10000 |
2.62B | 3.2578 |
step11000 |
2.88B | 3.2401 |
step12000 |
3.15B | 3.2227 |
step13000 |
3.41B | 3.2089 |
step14000 |
3.67B | 3.1942 |
step15000 |
3.93B | 3.1803 |
step16000 |
4.19B | 3.1676 |
step17000 |
4.46B | 3.1546 |
step18000 |
4.72B | 3.1423 |
step19000 |
4.98B | 3.1329 |
step20000 |
5.24B | 3.1220 |
step21000 |
5.51B | 3.1101 |
step22000 |
5.77B | 3.0984 |
step23000 |
6.03B | 3.0871 |
step24000 |
6.29B | 3.0769 |
step25000 |
6.55B | 3.0676 |
step26000 |
6.82B | 3.0563 |
step27000 |
7.08B | 3.0471 |
step28000 |
7.34B | 3.0387 |
step29000 |
7.60B | 3.0290 |
step30000 |
7.86B | 3.0214 |
step31000 |
8.13B | 3.0143 |
step32000 |
8.39B | 3.0067 |
step33000 |
8.65B | 2.9994 |
step34000 |
8.91B | 2.9947 |
step35000 |
9.18B | 2.9900 |
step36000 |
9.44B | 2.9857 |
step37000 |
9.70B | 2.9830 |
step38000 |
9.96B | 2.9808 |
step38139 |
10.00B | 2.9808 |
Usage
Requires transformers>=5.20.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JIRU-W/dense_e1_seq2048_gbs128_10b"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16) # final
ck = AutoModelForCausalLM.from_pretrained(repo, revision="step10000", trust_remote_code=True, dtype=torch.bfloat16)