qwen3-moe-tiny

A small (~670M parameter) Qwen3 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with prime-rl.

Fine-tuned on PrimeIntellect/Reverse-Text-SFT to provide a non-trivial distribution for KL divergence during RL.

Quick Start

uv run rl @ configs/ci/integration/rl_moe/qwen3_moe.toml

See the Testing MoE at Small Scale guide for full instructions.

Model Details

Links

• prime-rl - RL training framework
• PrimeIntellect - Building infrastructure for decentralized AI