Qwen3.8-27B MLX 4-bit for tqLLM (Apple Silicon)
The mlx-community 4-bit weights of Qwen3.8-27B, repacked into the GGUF layout tqLLM reads on Apple Silicon without requantising, plus z-lab's DFlash2 drafter in the same layout. Requires tqLLM 1.4.1 (tqllm.com). See docs/guides/mac.md.
| File | What |
|---|---|
Qwen3.8-27B-MLX-4bit.gguf |
the model (the same codes as mlx-community/Qwen3.8-27B-4bit) |
Qwen3.8-27B-DFlash2-MLX-4bit.gguf |
the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2), 4-bit |
Qwen3.8-27B-MLX-4bit-QSA.gguf |
the model plus the 16 QSA indexers of higa2515/Qwen3.8-27B-QSA-BF16 (FP16): sparse attention past 48K keys |
tqserve --model Qwen3.8-27B-MLX-4bit.gguf -md Qwen3.8-27B-DFlash2-MLX-4bit.gguf --context 70000
Decode tok/s on an M5 Pro 48 GB, one request, greedy, --exact on (every text equals plain one-token-at-a-time decoding):
| Context | 8K | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|
| tqLLM 1.4.1 | 51.9 | 62.1 | 56.1 | 48.8 | 38.9 | 27.5 |
| tqLLM 1.4.1, QSA file | dense | dense | dense | 45.7 | 49.6 | |
| TensorFold 1.2.1 | 61.0 | 54.4 | 49.9 | 41.4 | 34.7 | does not fit |
| Splash 1.4.0 | 60.8 | 62.4 | 48.4 | 43.9 | 35.9 | 24.5 |
QSA changes the text (spec output still equals the same file's plain decoding); --qsa-from N|off sets where it starts.
Credits: Qwen3.8-27B (Qwen), 4-bit quantisation by mlx-community, DFlash2 drafter by z-lab, QSA (arXiv 2608.30320) and the 27B indexers by higa2515 (Apache-2.0).