qwen3-moe-tiny
A small (~670M parameter) Qwen3 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with prime-rl.
Fine-tuned on PrimeIntellect/Reverse-Text-SFT to provide a non-trivial distribution for KL divergence during RL.
Quick Start
uv run rl @ configs/ci/integration/rl_moe/qwen3_moe.toml
See the Testing MoE at Small Scale guide for full instructions.
Model Details
| Parameter | Value |
|---|---|
| Hidden size | 1024 |
| Layers | 24 |
| Experts | 16 |
| Active experts | 4 |
| Parameters | ~670M |
Links
- prime-rl - RL training framework
- PrimeIntellect - Building infrastructure for decentralized AI