Qwen3.6-27B optimized for MLX.
- A mixed-precision quant that balances speed, memory, and accuracy.
- 4-bit baseline with important layers at 6, 8, and BF16.
- This quant supports image input and requires a vision-capable server. Non-vision version here.
Usage
# Start server at http://localhost:8080/v1/chat/completions
uvx --from mlx-vlm \
mlx_vlm.server \
--host 127.0.0.1 \
--port 8080 \
--model spicyneuron/Qwen3.6-27B-MLX-5.7bit-vision
Benchmarks
| metric | unsloth/Qwen3.6-27B-UD-MLX-4bit | mlx-community/Qwen3.6-27B-OptiQ-4bit | 5.7 bit (this model) |
|---|---|---|---|
| bpw | 7.516 | 5.575 | 5.679 |
| base memory | 23.534 | 17.457 | 17.781 |
| peak memory (1024/512) | 27.085 | 20.633 | 20.966 |
| prompt tok/s (1024) | 420.712 ± 0.129 | 428.184 ± 0.165 | 422.521 ± 0.948 |
| gen tok/s (512) | 24.759 ± 0.025 | 31.521 ± 0.030 | 30.460 ± 0.106 |
| kl mean | 0.031 ± 0.003 | 0.044 ± 0.004 | 0.027 ± 0.002 |
| kl p95 | 0.107 ± 0.003 | 0.164 ± 0.004 | 0.103 ± 0.002 |
| perplexity* | 4.560 ± 0.026 | 4.850 ± 0.020 | 4.872 ± 0.029 |
| hellaswag | 0.552 ± 0.011 | 0.552 ± 0.011 | 0.556 ± 0.011 |
Unsloth's "4bit" actually averages 7.5 per weight even after excluding the vision tower. This quant is smaller, matches in KL divergence and Hellaswag, and has significantly faster token generation.
OptiQ lands around the same size. This quant is slightly slower but slightly better on KLD (measured against this dataset).
* Perplexity on this model seems to swing a ton based on number of samples, so treat this as a noisy result.
Tested on a Mac Studio M3 Ultra with:
mlx_lm.convert --hf-path Qwen/Qwen3.6-35B-A3B --mlx-path ./mlx && mlx_lm.kld --baseline-model ./mlx
mlx_lm.perplexity --sequence-length 1024 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 2000
Required PRs:
Methodology
Quantized with a mlx-vlm fork. MLX quantization options differ than llama.cpp, but the principles are the same:
- Sensitive layers like MoE routing, attention, and output embeddings get higher precision
- More tolerant layers like MoE experts get lower precision