𓌳 REAP𓌳 the Experts: Why Pruning Prevails for One-Shot MoE Compression
Introducing Step-3.5-Flash-REAP-149B-A11B, a memory-efficient compressed variant of Step-3.5-Flash that maintains near-identical performance while being 25% lighter.
This model was created using REAP (Router-weighted Expert Activation Pruning), a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
Step-3.5-Flash-REAP-149B-A11B has the following specifications:
| Benchmark | Step-3.5-Flash | Step-3.5-Flash-REAP-149B-A11B | Step-3.5-Flash-REAP-121B-A11B | |
|---|---|---|---|---|
| Compression | — | 25% | 40% | |
| Coding | ||||
| HumanEval | 98.2 | 97.0 | 95.7 | |
| HumanEval+ | 93.9 | 90.9 | 91.5 | |
You can deploy the model directly using the latest vLLM (that supports Step-3.5-Flash), no source modifications or custom patches required.
vllm serve cerebras/Step-3.5-Flash-REAP-149B-A11B \
--tensor-parallel-size 8 \
--tool-call-parser step3p5 \
--reasoning-parser step3p5 \
--trust-remote-code \
--enable_expert_parallel \
--disable-cascade-attn \
--enable-auto-tool-choice
If you encounter insufficient memory when running this model, you might need to set a lower value for --max-num-seqs flag (e.g. set to 64). For more information, refer to the official vLLM deployment guide.
This checkpoint was created by applying the REAP (Router-weighted Expert Activation Pruning) method uniformly across all Mixture-of-Experts (MoE) blocks of Step-3.5-Flash, with a 25% pruning rate.
REAP selects experts to prune based on a novel saliency criterion that considers both:
This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
📚 For more details, refer to the following resources:
This model is derived from
stepfun-ai/Step-3.5-Flash
and distributed under the Apache 2.0 license.
If you use this checkpoint, please cite the REAP paper:
@article{lasby-reap,
title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
journal={arXiv preprint arXiv:2510.13999},
year={2025}
}
model-00001-of-00064.safetensors 5.0 GBmodel-00013-of-00064.safetensors 4.8 GBmodel-00016-of-00064.safetensors 4.8 GBmodel-00017-of-00064.safetensors 4.8 GBmodel-00019-of-00064.safetensors 4.8 GBmodel-00022-of-00064.safetensors 4.8 GBmodel-00023-of-00064.safetensors 4.8 GBmodel-00025-of-00064.safetensors 4.8 GBmodel-00028-of-00064.safetensors 4.8 GBmodel-00029-of-00064.safetensors 4.8 GBmodel-00031-of-00064.safetensors 4.8 GBmodel-00034-of-00064.safetensors 4.8 GBmodel-00035-of-00064.safetensors 4.8 GBmodel-00037-of-00064.safetensors 4.8 GBmodel-00040-of-00064.safetensors 4.8 GBmodel-00041-of-00064.safetensors 4.8 GBmodel-00043-of-00064.safetensors 4.8 GBmodel-00046-of-00064.safetensors 4.8 GBmodel-00047-of-00064.safetensors 4.8 GBmodel-00049-of-00064.safetensors 4.8 GBmodel-00052-of-00064.safetensors 4.8 GBmodel-00053-of-00064.safetensors 4.8 GBmodel-00055-of-00064.safetensors 4.8 GBmodel-00058-of-00064.safetensors 4.8 GBmodel-00059-of-00064.safetensors 4.8 GBmodel-00061-of-00064.safetensors 4.8 GBmodel-00004-of-00064.safetensors 4.8 GBmodel-00005-of-00064.safetensors 4.8 GBmodel-00007-of-00064.safetensors 4.8 GBmodel-00010-of-00064.safetensors 4.8 GBmodel-00011-of-00064.safetensors 4.8 GBmodel-00014-of-00064.safetensors 4.7 GBmodel-00020-of-00064.safetensors 4.7 GBmodel-00026-of-00064.safetensors 4.7 GBmodel-00032-of-00064.safetensors 4.7 GBmodel-00038-of-00064.safetensors 4.7 GBmodel-00044-of-00064.safetensors 4.7 GBmodel-00050-of-00064.safetensors 4.7 GBmodel-00056-of-00064.safetensors 4.7 GBmodel-00062-of-00064.safetensors 4.7 GBmodel-00002-of-00064.safetensors 4.7 GBmodel-00008-of-00064.safetensors 4.7 GBmodel-00003-of-00064.safetensors 4.5 GBmodel-00006-of-00064.safetensors 4.5 GBmodel-00009-of-00064.safetensors 4.5 GBmodel-00012-of-00064.safetensors 4.5 GBmodel-00015-of-00064.safetensors 4.5 GBmodel-00018-of-00064.safetensors 4.5 GBmodel-00021-of-00064.safetensors 4.5 GBmodel-00024-of-00064.safetensors 4.5 GBmodel-00027-of-00064.safetensors 4.5 GBmodel-00030-of-00064.safetensors 4.5 GBmodel-00033-of-00064.safetensors 4.5 GBmodel-00036-of-00064.safetensors 4.5 GBmodel-00039-of-00064.safetensors 4.5 GBmodel-00042-of-00064.safetensors 4.5 GBmodel-00045-of-00064.safetensors 4.5 GBmodel-00048-of-00064.safetensors 4.5 GBmodel-00051-of-00064.safetensors 4.5 GBmodel-00054-of-00064.safetensors 4.5 GBmodel-00057-of-00064.safetensors 4.5 GBmodel-00060-of-00064.safetensors 4.5 GBmodel-00063-of-00064.safetensors 4.5 GBmodel-00064-of-00064.safetensors 3.4 GBHave the files? Submit a torrent.
Submit a torrent