glm4-moe-tiny
A small (~543M parameter) GLM-4 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with prime-rl.
Fine-tuned on PrimeIntellect/Reverse-Text-SFT to provide a non-trivial distribution for KL divergence during RL.
Quick Start
uv run rl @ configs/ci/integration/rl_moe/glm4_moe.toml
See the Testing MoE at Small Scale guide for full instructions.
Model Details
| Parameter | Value |
|---|---|
| Hidden size | 1024 |
| Layers | 24 |
| Experts | 8 |
| Active experts | 4 |
| Parameters | ~543M |
Links
- prime-rl - RL training framework
- PrimeIntellect - Building infrastructure for decentralized AI