Instella-MoE is a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters trained end-to-end from pre-training to RL. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs using AMD's Primus framework, Instella-MoE combines a sparsely activated MoE design with architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective.
Figure 1: Pre-trained and Post-trained Instella-MoE model performance compared with other similar size state-of-the-art models.
The release includes checkpoints from the following Instella-MoE training pipeline stages as shown in Table 1 below:
| Model | Stage | Description |
|---|---|---|
| Instella-MoE-16B-A3B-Pretrain (Link) | Pre-training | MoE base model trained from scratch on a large and diverse training corpus. |
| Instella-MoE-16B-A3B-Midtrain (Link) | Mid-training | Pretrained model further trained on high-quality data mixtures to refine key capabilities. |
| Instella-MoE-16B-A3B-Base (Link) | Long-context | Long-context training to extend the model’s ability to process and reason over longer sequences. We use this as our final base checkpoint. |
| Instella-MoE-16B-A3B-SFT (Link) | SFT | Base checkpoint extended via supervised fine-tuning (SFT) to enable instruction following and chain-of-thought reasoning capabilities. |
| Instella-MoE-16B-A3B-DPO (Link) | DPO | Direct preference optimization (DPO) on contrastive preference data to improve model performance. |
| Instella-MoE-16B-A3B-Think (Link) | RL | Final thinking checkpoint refined with reinforcement learning (RL) to further strengthen instruction following and overall response quality. |
Table 1: Instella-MoE-16B-A3B models and training stages.
| Parameter | Value |
|---|---|
| Total Parameters | 16B |
| Active Parameters per Token | 2.8B |
| Decoder Layers | 27 |
| Hidden Size | 2048 |
| Attention Heads | 16 |
| Number of Experts | 64 |
| Shared Experts | 2 |
| Activated Experts per Token | 6 |
| Vocabulary Size | 128,896 |
| Attention | Gated Multi-head Latent Attention (Gated MLA) |
| MoE Connectivity | FarSkip-Collective |
Table 2: Instella-MoE-16B-A3B-Base Results on Standard Benchmarks.
Table 3: Instella-MoE-16B-A3B-Base Results on Long Context HELMET and RULER Benchmarks.
Table 4: Instella-MoE-Think results. We evaluate all models using the OLMES framework, generating up to a maximum of 32768 tokens.
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-MoE-16B-A3B-Think"
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)
prompt = [{"role": "user", "content": "What are the computational benefits of Mixture-of-Experts models?"}]
inputs = tokenizer.apply_chat_template(
prompt,
add_generation_prompt=True,
return_tensors='pt'
)
tokens = model.generate(
inputs.to(model.device),
max_new_tokens=1024,
temperature=0.6,
top_p=0.95,
do_sample=True
)
print(tokenizer.decode(tokens[0], skip_special_tokens=False))
For high-throughput inference with SGLang, please refer to the setup and usage instructions in our GitHub repository.
Instella-MoE was trained end-to-end on AMD Instinct™ MI300X and MI325X GPUs using the AMD ROCm™ software stack, built on top of the Primus training framework and the Miles RL framework. Training proceeds through a multi-stage pipeline—pre-training, mid-training, long-context extension, SFT, DPO, and RL—with each stage progressively strengthening the model's capabilities.
For the complete training recipe, including per-stage data mixtures, hyperparameters, training frameworks, and inference code, please refer to our GitHub repository and technical blog.
We are deeply grateful to the LLM360 team and the Miles team for their invaluable support throughout the development of our model.
Core contributors: Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Zicheng Liu
Contributors: Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Emad Barsoum
If you use Instella-MoE, please cite:
@article{instella-moe,
title={Instella-MoE Technical Report},
author={Liu, Jiang and Ranjan, Sudhanshu and Mishra, Prakamya and Dukler, Yonatan and Ramesh, Gowtham and Wu, Jialian and Sun, Ximeng and Xie, Wen and Hou, Chaojun and Appia, Vikram and Gu, Zhenyu and Liu, Zicheng and Barsoum, Emad},
journal={arXiv preprint arXiv:2609.00791},
year={2026}
}
@inproceedings{
dukler2026farskipcollective,
title={FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models},
author={Yonatan Dukler and Guihong Li and Deval Shah and Jiang Liu and Vikram Appia and Emad Barsoum},
booktitle={Ninth Conference on Machine Learning and Systems},
year={2026},
url={https://openreview.net/forum?id=ruOpvLzsGV}
}
model-00001-of-00006.safetensors 5.4 GBmodel-00003-of-00006.safetensors 5.4 GBmodel-00004-of-00006.safetensors 5.4 GBmodel-00005-of-00006.safetensors 5.4 GBmodel-00002-of-00006.safetensors 5.4 GBmodel-00006-of-00006.safetensors 4.9 GB有文件?提交种子。
提交种子