Qwen3.5-0.8B-MoE-A0.6B
The text model of Qwen/Qwen3.5-0.8B, converted from dense to a Mixture-of-Experts without pretraining from scratch. Every MLP is split into 28 experts of 128 neurons, and a router picks 14 of them per token, which halves the MLP compute. The original model was then distilled into the result, including its chat format.
Architecture
| Qwen3.5-0.8B (dense, text) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| Architecture | Qwen3_5ForConditionalGeneration (text part) |
Qwen3_5MoeForCausalLM |
| Layers | 24: 18 Gated DeltaNet + 6 gated full attention (3:1) | same, unchanged |
| Hidden size / vocab | 1024 / 248,320 (tied embeddings) | same |
| MLP per layer | 1 x 3584 SwiGLU | 28 experts x 128, top-14 (trained at top-12) |
| Total parameters | 752.4M | 755.5M (+0.7M routers, +2.4M zero shared expert) |
| Active parameters per token | 752.4M | ~623M (~369M excluding embeddings / LM head) |
| Active MLP compute | 100% | 50% |
Evaluation
| Qwen3.5-0.8B (dense) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| Wikitext-2 ppl ↓ | 19.38 | 21.90 (+13.0%) |
| C4 ppl ↓ | 24.66 | 28.37 (+15.0%) |
| Qwen3.5-0.8B (dense) | Qwen3.5-0.8B (MoE) | |
|---|---|---|
| HellaSwag | 0.488 | 0.450 |
| ARC-Easy | 0.629 | 0.545 |
| ARC-Challenge | 0.370 | 0.287 |
| WinoGrande | 0.562 | 0.562 |
| OpenBookQA | 0.306 | 0.294 |
| Task mean | 0.471 | 0.428 |
How to run
transformers
Tested with transformers 5.5.0. For fast Gated DeltaNet kernels, also install fla-core. Without it,
transformers falls back to a slower pure-PyTorch path.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "<your-username>/Qwen3.5-0.8B-MoE-E28-A0.6B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "Write one sentence about the sea."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.8)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
Usage notes:
- Thinking: keep thinking mode off. It is the template's default; it was never trained after conversion.
- Context: distillation used 512-token sequences and evaluation 1024; longer contexts were not checked.
- Facts: don't rely on it for facts.
Support
If you find this work useful or interesting, any contribution helps.
License
Apache 2.0, the same as the base model Qwen/Qwen3.5-0.8B.