Qwen3.5-0.8B-MoE-A0.6B
The text model of Qwen/Qwen3.5-0.8B, converted from dense to a Mixture-of-Experts without pretraining from scratch. Every MLP is split into 28 experts of 128 neurons, and a router picks 14 of them per token, which halves the MLP compute. The original model was then distilled into the result, including its chat format.
Architecture
| Qwen3.5-0.8B (dense, text) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| Architecture | Qwen3_5ForConditionalGeneration (text part) |
Qwen3_5MoeForCausalLM |
| Layers | 24: 18 Gated DeltaNet + 6 gated full attention (3:1) | same, unchanged |
| Hidden size / vocab | 1024 / 248,320 (tied embeddings) | same |
| MLP per layer | 1 x 3584 SwiGLU | 28 experts x 128, top-14 (trained at top-12) |
| Total parameters | 752.4M | 755.5M (+0.7M routers, +2.4M zero shared expert) |
| Active parameters per token | 752.4M | ~623M (~369M excluding embeddings / LM head) |
| Active MLP compute | 100% | 50% |
Evaluation
| Qwen3.5-0.8B (dense) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| Wikitext-2 ppl ↓ | 19.38 | 21.90 (+13.0%) |
| C4 ppl ↓ | 24.66 | 28.37 (+15.0%) |
| Qwen3.5-0.8B (dense) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| HellaSwag | 0.488 | 0.450 |
| ARC-Easy | 0.629 | 0.545 |
| ARC-Challenge | 0.370 | 0.287 |
| WinoGrande | 0.562 | 0.562 |
| OpenBookQA | 0.306 | 0.294 |
| Task mean | 0.471 | 0.428 |
llama.cpp
The GGUFs use llama.cpp's built-in qwen35moe architecture, so they need no patches. They were tested with a local
CUDA build (b11243).
llama-cli -m qwen3.5-0.8b-moe-e28k12-chat-Q4_K_M.gguf -cnv --temp 0.7 --top-p 0.8
llama-server -m qwen3.5-0.8b-moe-e28k12-chat-Q4_K_M.gguf -c 4096
Usage notes:
- Thinking: keep thinking mode off. It is the template's default; it was never trained after conversion.
- Context: distillation used 512-token sequences and evaluation 1024; longer contexts were not checked.
- Facts: don't rely on it for facts.
Support
If you find this work useful or interesting, any contribution helps.
License
Apache 2.0, the same as the base model Qwen/Qwen3.5-0.8B.