muon-PT-adamw-MT-a10-g20-fdr
The Muon PT checkpoint after 10,000 steps (20.97B tokens) of midtraining with AdamW on DCLM and FineMath-4plus. FineMath accounts for 10% of the token budget. The final 5,000 steps use 20% FineMath. The learning rate decays linearly from 5e-4 to 1e-5 over all 10,000 steps.
Final 32M-token subset losses were 2.7294 on DCLM and 1.6516 on FineMath. Reproduction of arXiv 2610.29348 with Megatron-LM.