Qwen3-0.6B-MoE-A0.5B-GGUF

GGUF files of Qwen3-0.6B-MoE-A0.5B: Qwen/Qwen3-0.6B converted into a Mixture-of-Experts (32 experts of 96 neurons per layer, top-16 routing) and distilled from the original, including its chat format. See the main model card for how it was made and for its limitations. The most important ones: use non-thinking mode, and don't rely on it for facts.

The files use llama.cpp's own qwen3moe architecture, so they run in stock llama.cpp and anything built on it, with no patches.

Files

Perplexities are from llama-perplexity on a 40 x 512-token wikitext-2 sample, so they are lower than the 1024-token numbers on the main card. The dense Qwen3-0.6B loses 7.9% at Q4_K_M on the same sample, so the MoE quantises no worse than the original. In the Q4_K_M file, the expert down projections (96 wide, not a multiple of 256) use Q5_0 / Q8_0 instead of K-quants.

Usage

llama-cli -m Qwen3-0.6B-MoE-A0.5B-Q4_K_M -cnv -p "You are a helpful assistant." \
  --temp 0.7 --top-p 0.8

llama-server -m Qwen3-0.6B-MoE-A0.5B-Q4_K_M -c 2048

The GGUF carries Qwen3's chat template. Keep thinking off: add /no_think to your messages, or use a frontend option that disables thinking.

Speed

llama-bench, Q4_K_M, RTX 4060 (8 GB), tokens per second:

Token generation is 7% slower than dense on GPU and 8% faster on CPU. Prompt processing is slower, because llama.cpp's MoE kernels are less efficient on large batches. At this size, attention and the embeddings dominate the work per token, so a larger saving needs a larger model.

Compatibility

• Architecture qwen3moe: all 28 layers MoE, no shared expert, softmax top-k routing renormalised over the chosen experts (llama.cpp's built-in behaviour for this architecture), expert_used_count 16.
• Tested with llama.cpp built locally on Windows (CUDA). Any recent llama.cpp release with qwen3moe support should load it.

Support

If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen3-0.6B.