Qwen2.5-1.5B-MoE-A0.9B-GGUF

GGUF files of Qwen2.5-1.5B-MoE-A0.9B, a Mixture-of-Experts conversion of Qwen2.5-1.5B with ~0.85B active parameters per token instead of 1.54B. They run in stock llama.cpp and anything built on it (LM Studio, Ollama, Jan, ...). No patches are needed.

In llama.cpp it generates text 32% faster than the dense Qwen2.5-1.5B on GPU and 41% faster on CPU, at the cost of about 40% higher perplexity. See the main model card for the architecture and full benchmark results.

This is a base (completion) model. It is not instruction-tuned.

Files

Use BF16, not F16, as the full-precision format: some activations in this model exceed the F16 range.

Quick start

llama.cpp

llama-cli -hf YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M -p "The history of the printing press" -n 128
llama-server -hf YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M

Ollama

ollama run hf.co/YOUR_USERNAME/Qwen2.5-1.5B-MoE-A0.9B-GGUF:Q4_K_M

LM Studio: search for Qwen2.5-1.5B-MoE-A0.9B-GGUF and download the Q4_K_M file.

Fidelity

llama.cpp llama-perplexity against the original transformers model on the same 40 × 512 tokens of WikiText-2:

Speed

llama-bench, Q4_K_M for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU):

The Q4_K_M file is larger than the dense model's (1125 vs 935 MiB) because llama.cpp's qwen2moe needs a separate output head rather than one tied to the embeddings.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen2.5-1.5B.