Qwen2.5-1.5B-MoE-A0.9B-GGUF
GGUF files of Qwen2.5-1.5B-MoE-A0.9B, a Mixture-of-Experts conversion of Qwen2.5-1.5B with ~0.85B active parameters per token instead of 1.54B. They run in stock llama.cpp and anything built on it (LM Studio, Ollama, Jan, ...). No patches are needed.
In llama.cpp it generates text 32% faster than the dense Qwen2.5-1.5B on GPU and 41% faster on CPU, at the cost of about 40% higher perplexity. See the main model card for the architecture and full benchmark results.
This is a base (completion) model. It is not instruction-tuned.
Files
| File | Size | Use |
|---|---|---|
Qwen2.5-1.5B-MoE-A0.9B-Q4_K_M.gguf |
1125 MiB | Recommended for most uses |
Qwen2.5-1.5B-MoE-A0.9B-BF16.gguf |
3401 MiB | Full precision, or a source for other quantizations |
Use BF16, not F16, as the full-precision format: some activations in this model exceed the F16 range.
Quick start
llama.cpp
llama-cli -hf YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M -p "The history of the printing press" -n 128
llama-server -hf YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M
Ollama
ollama run hf.co/YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M
LM Studio: search for Qwen2.5-1.5B-MoE-A0.9B-GGUF and download the Q4_K_M file.
Fidelity
llama.cpp llama-perplexity against the original transformers model on the same 40 × 512 tokens of WikiText-2:
| Perplexity | vs transformers | |
|---|---|---|
| transformers (bf16) | 12.258 | |
| BF16 GGUF | 12.270 | +0.10% |
| Q4_K_M GGUF | 12.683 | +3.5% |
Speed
llama-bench, Q4_K_M for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU):
| Dense Qwen2.5-1.5B | This model | Change | |
|---|---|---|---|
| GPU, text generation | 191 | 253 | +32% |
| GPU, prompt processing (512 tokens) | 10,329 | 11,353 | +10% |
| CPU, text generation | 54.9 | 77.5 | +41% |
| CPU, prompt processing (512 tokens) | 3,615 | 3,646 | ≈ same |
The Q4_K_M file is larger than the dense model's (1125 vs 935 MiB) because llama.cpp's qwen2moe needs a separate
output head rather than one tied to the embeddings.
Support
This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.
License
Apache 2.0, the same as the base model Qwen/Qwen2.5-1.5B.