This is the text model of Qwen/Qwen3.5-2B, converted from dense to a Mixture-of-Experts without pretraining. Every MLP is split into 32 experts of 192 neurons, and a router picks 12 per token, so the model runs 37.5% of the original MLP compute.
| Qwen3.5-2B (dense, text) | This model | |
|---|---|---|
| Layers | 24 (Gated DeltaNet + gated full attention) | same, unchanged |
| Hidden size / vocab | 2048 / 248,320 (tied embeddings) | same |
| MLP per layer | 1 × 6144 SwiGLU | 32 experts × 192, top-12 |
| Total parameters | 1.88B | 1.89B |
| Active parameters per token | 1.88B | 1.32B (813M excluding embeddings) |
| Active MLP compute | 100% | 37.5% |
| Qwen3.5-2B (dense) | This model | |
|---|---|---|
| Wikitext-2 ppl ↓ | 13.60 | 14.68 (+7.9%) |
| C4 ppl ↓ | 18.73 | 25.29 (+35%) |
| Qwen3.5-2B (dense) | This model | |
|---|---|---|
| HellaSwag | 0.565 | 0.497 |
| ARC-Easy | 0.679 | 0.664 |
| ARC-Challenge | 0.413 | 0.370 |
| WinoGrande | 0.623 | 0.579 |
| OpenBookQA | 0.350 | 0.324 |
| Task mean | 0.526 | 0.487 |
| Fact recall (287 facts) | 0.948 | 0.760 |
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "<your-username>/Qwen3.5-2B-MoE-E32-A1.3B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "Write one sentence about the sea."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.8)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
Usage notes:
transformers 5.5.0. Install fla-core for fast DeltaNet kernels.If you find this work useful or interesting, any contribution helps.
Apache 2.0, the same as the base model.
Qwen/Qwen3.5-2B