Qwen3.5 9B optimized to run on Mac.
- A mixed-precision quant that balances speed, memory, and accuracy.
- 4-bit baseline with important layers at 8 and BF16.
Usage
# Start server at http://localhost:8080/chat/completions
uvx --from mlx-lm --with torchvision \
mlx_vlm.server \
--host 127.0.0.1 \
--port 8080 \
--model spicyneuron/Qwen3.5-9B-MLX-5.6bit-vision
Benchmarks
| metric | this model |
|---|---|
| bpw | 5.554 |
| base memory | 5.789 |
| peak memory (1024/512) | 7.446 |
| prompt tok/s (1024) | 1481.661 ± 6.709 |
| gen tok/s (512) | 91.086 ± 0.101 |
| kl mean | 0.032 ± 0.002 |
| kl p95 | 0.069 ± 0.002 |
| perplexity | 3.739 ± 0.018 |
| winogrande | 0.660 ± 0.021 |
Tested on a Mac Studio M3 Ultra. KL divergence is approximate, based on top_k not full logits. Here's the code.
mlx_lm.kld --baseline-model path/to/mlx-full-precision
mlx_lm.perplexity --sequence-length 2048 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 500
Methodology
Quantized with a mlx-lm fork, drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. MLX quantization options differ from llama.cpp, but the principles are the same:
- Sensitive layers like MoE routing, attention, and output embeddings get higher precision
- More tolerant layers like MoE experts get lower precision