Doge 120M MoE checkpoint

Doge uses wsd_scheduler as the training scheduler, which divides the learning rate into three stages: warmup, stable, and decay. It allows us to continue training on any new dataset from any checkpoint in the stable stage without spikes of the training.

Here are the initial learning rates required to continue training at each checkpoint:

• Doge-20M: 8e-3
• Doge-20M-MoE: 8e-3
• Doge-60M: 6e-3
• Doge-120M-MoE: 6e-3
• Doge-160M: 4e-3
• Doge-480M-MoE: 4e-3
• Doge-320M: 2e-3
• Doge-1.4B-MoE: 2e-3