This is the text model of Qwen/Qwen3.5-2B, converted from dense to a
Mixture-of-Experts without pretraining. Every MLP is split into 32 experts of 192 neurons, and a router picks 12 per
token, so the model runs 37.5% of the original MLP compute. It loads as a stock Qwen3_5MoeForCausalLM and runs in stock
llama.cpp (qwen35moe). GGUF files are included.
| Qwen3.5-2B (dense, text) | This model | |
|---|---|---|
| Layers | 24 (Gated DeltaNet + gated full attention) | same, unchanged |
| Hidden size / vocab | 2048 / 248,320 (tied embeddings) | same |
| MLP per layer | 1 × 6144 SwiGLU | 32 experts × 192, top-12 |
| Total parameters | 1.88B | 1.89B |
| Active parameters per token | 1.88B | 1.32B (813M excluding embeddings) |
| Active MLP compute | 100% | 37.5% |
| Qwen3.5-2B (dense) | This model | |
|---|---|---|
| Wikitext-2 ppl ↓ | 13.60 | 14.68 (+7.9%) |
| C4 ppl ↓ | 18.73 | 25.29 (+35%) |
| Qwen3.5-2B (dense) | This model | |
|---|---|---|
| HellaSwag | 0.565 | 0.497 |
| ARC-Easy | 0.679 | 0.664 |
| ARC-Challenge | 0.413 | 0.370 |
| WinoGrande | 0.623 | 0.579 |
| OpenBookQA | 0.350 | 0.324 |
| Task mean | 0.526 | 0.487 |
| Fact recall (287 facts) | 0.948 | 0.760 |
GGUF files:
| File | Size | ppl, llama.cpp vs transformers | Speed on RTX 4060 (tg128 / pp512) |
|---|---|---|---|
q35-2b-k12h2-bbtm-bf16.gguf |
3616 MiB | −0.24% | not measured |
q35-2b-k12h2-bbtm-Q4_K_M.gguf |
1277 MiB | +3.17% | 177 / 6009 t/s |
The ppl comparison uses llama-perplexity on 40 × 512 tokens of wikitext-2.
llama-cli -m Qwen35-2B-MoE-A1.3B-bf16.gguf -cnv --temp 0.7 --top-p 0.8
Usage notes:
transformers 5.5.0. Install fla-core for fast DeltaNet kernels.If you find this work useful or interesting, any contribution helps.
Apache 2.0, the same as the base model.