Qwen2.5-1.5B-MoE-A0.9B

A Mixture-of-Experts version of Qwen2.5-1.5B, converted from the dense model without pre-training from scratch. Every MLP layer was split into an always-on shared expert plus 32 routed experts, and each token uses 8 of them. That leaves ~0.85B active parameters per token instead of 1.54B. Every layer is MoE, so the model runs in stock llama.cpp with no patches. In llama.cpp it generates text 32% faster than the dense model on GPU and 41% faster on CPU.

GGUF files (llama.cpp, LM Studio, Ollama): Qwen2.5-1.5B-MoE-A0.9B-GGUF

This is a base (completion) model, like the original. It is not instruction-tuned.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "bayliner1980/Qwen2.5-1.5B-MoE-A0.9B"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16", device_map="auto")

ids = tok("The history of the printing press", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=80, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

It loads as a standard Qwen2MoeForCausalLM; no custom code or trust_remote_code is needed.

Architecture

* Active parameters exclude the embedding lookup and include the output head. The total grows only because the output head is untied; the experts together hold the same number of neurons as the original MLP.

The routing uses norm_topk_prob: false, the only mode llama.cpp's qwen2moe supports. The expert width (224) is a multiple of 32, so every expert tensor quantizes with llama.cpp's k-quants.

Quality

Perplexity: lower is better. Tasks: zero-shot acc_norm, higher is better, on the first 1000 items of each benchmark (500 for OpenBookQA), with the dense model scored the same way.

Speed

llama.cpp llama-bench, Q4_K_M for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU):

The GGUF files are larger than the dense model's (Q4_K_M: 1125 vs 935 MiB) because of the untied output head.

How it was made

The dense model's MLP neurons were grouped into a shared expert and routed experts, a router was fitted for each layer. Everything ran on a single 8 GB consumer GPU. No layer was kept dense, so the model fits llama.cpp's qwen2moe architecture as is.

Limitations

• About 41–45% higher perplexity than the dense original, and clearly weaker on reasoning benchmarks (ARC in particular). Use it where speed matters more than the last bit of quality.
• A base model: it continues text and does not follow instructions or chat reliably.
• Distilled on English educational web text only. Other languages and code may have degraded more than the English numbers suggest.
• Inherits the biases and limitations of Qwen2.5-1.5B.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen2.5-1.5B.