muon-PT
494M Qwen2.5-0.5B-shaped model with tied embeddings and a 4,096-token context, pretrained from scratch on 200B DCLM-Baseline tokens (95,368 steps, batch 512 × 4,096). Muon was used for the transformer matrices and AdamW for the remaining parameters. Final full-validation losses were 2.8162 on DCLM and 2.1709 on FineMath.
Part of a reproduction of Data Schedules and Learning Rate Schedules Should Be Designed Together (arXiv 2610.29348) with Megatron-LM. Converted with Megatron-Bridge's Qwen2 mappings.