Qwen2.5-0.5B-MoE-A0.3B-GGUF
A GGUF version of my MoE-Ified Qwen/Qwen2.5-0.5B. This model was quantized using llama.cpp. This model had to be modified to be compatible with llama.cpp so there was a decrease in quality.
Support
This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.
Files
| File | Size | Use |
|---|---|---|
Qwen2.5-0.5B-MoE-A0.3B-Q4_K_M.gguf |
487 MiB | llama.cpp, LM Studio, Ollama. Recommended. |
Qwen2.5-0.5B-MoE-A0.3B-BF16.gguf |
1210 MiB | llama.cpp at full precision, or as a source for other quantizations |
model.safetensors + configs |
1204 MiB | Hugging Face transformers (Qwen2MoeForCausalLM) |
Use the BF16 GGUF, not F16. Some activations exceed the F16 range, and an F16 GGUF produces NaNs.
Architecture
| Qwen2.5-0.5B (dense) | This model | |
|---|---|---|
| Layers / hidden size / heads | 24 / 896 / 14 (2 KV) | same |
| MLP per layer | 4864 neurons, all active | shared expert 768 + 32 experts × 128, top-8 |
| Active MLP neurons per token | 4864 | 1792 (37%) |
| Active parameters per token* | 494M | 297M |
| Total parameters | 494M | 631M |
| Output head | tied to embeddings | untied (llama.cpp's qwen2moe requires a separate output tensor) |
* Active parameters exclude the embedding lookup and include the output head. The total grows only because the output head is untied. The experts together hold exactly the original MLP's neurons.
The routing is configured with norm_topk_prob: false, the only mode llama.cpp's qwen2moe supports. The
renormalisation the model was originally trained with is folded into the expert weights.
Quality
Measured with this repository's evaluation scripts. Perplexity is lower-is-better; tasks are mean acc_norm
(higher is better).
| Dense Qwen2.5-0.5B | This model | |
|---|---|---|
| WikiText-2 test perplexity (1024-token windows) | 14.66 | 20.00 (+36%) |
| C4 validation perplexity (64 × 1024 tokens) | 21.29 | 30.10 (+41%) |
| Zero-shot tasks, mean acc_norm | 0.467† | 0.411 |
† Full benchmark sets. The MoE score uses the first 1000 items of each (500 for OpenBookQA).
Per-task (first 1000 items, acc_norm): HellaSwag 0.449, ARC-Easy 0.498, ARC-Challenge 0.252, WinoGrande 0.541, OpenBookQA 0.316.
GGUF fidelity (llama.cpp llama-perplexity vs transformers on the same 40 × 512-token WikiText-2 chunks):
| Perplexity | vs transformers | |
|---|---|---|
| transformers (bf16) | 16.772 | |
| BF16 GGUF | 16.763 | −0.05% |
| Q4_K_M GGUF | 17.068 | +1.8% |
Speed (Q4_K_M, llama.cpp, vs the dense Qwen2.5-0.5B at the same quantization): token generation is about 24% faster on CPU and 6% faster on GPU. CPU prompt processing is about 11% slower. A model this small is dominated by attention and the 151k-token output head, so the savings grow with larger models.
Limitations
- About 36–41% higher perplexity than the dense original, and somewhat weaker on reasoning benchmarks. Use this model where speed or experimentation matters more than the last bit of quality.
- A base model: it continues text and does not follow instructions or chat reliably.
- Distilled mostly on English educational web text. Other languages and code may have degraded more than the English numbers suggest.
- Inherits the biases and limitations of Qwen2.5-0.5B.
License
Apache 2.0, the same as the base model Qwen/Qwen2.5-0.5B.