muon-PT-muon-MT-a10-g100-mdr
The Muon PT checkpoint after 10,000 steps (20.97B tokens) of midtraining with Muon on DCLM and FineMath-4plus. FineMath accounts for 10% of the token budget. The final 1,000 steps use 100% FineMath. The learning rate stays at 5e-4 for 9,000 steps, then decays linearly to 1e-5.
Final 32M-token subset losses were 2.9684 on DCLM and 1.5509 on FineMath. Reproduction of arXiv 2610.29348 with Megatron-LM.