d24 SFT - v2 base + OLMo-3 (5B chunked) midtrain (0.757B)
nanochat-style depth-24 decoder (24L x 1536 hidden x 12 heads, SwiGLU / RoPE / RMSNorm,
tied embeddings, GPT-2 BPE vocab 50304, 0.757B params, 2048 ctx, LlamaForCausalLM).
Lineage: v2 pretrain (13.1B ClimbMix, WSD) -> OLMo-3 Dolmino (5B, chunked) midtrain
(5.24B tok, 1 epoch, sfanm/d24-midtrain-olmo3-5b - the full OLMo-3 mix incl. reasoning traces +
olmOCR PDFs, kept by chunking long docs) -> SFT (nanochat chat mix, 1 epoch, sfanm/d24-sft-mixture).
Eval (greedy, full sets): GSM8K 4.62% | MMLU 35.1% | ARC 37.0%.
Beats the 2.3B-midtrain SFT (sfanm/d24-sft-v2-olmo3-2.3B: 3.79 / 32.8 / 35.6) on all three -
the larger, fuller OLMo-3 mix improves both math and general capability.
Chat format: ChatML (<|im_start|>role\n...<|im_end|>); see chat_template.jinja.