Spark-X2.6-4B GPTQ 4-bit (MLX)
MLX quantization of XHToken/Spark-X2.6-4B for Apple Silicon.
- All 180 linear layers quantized to 4-bit with GPTQ (group size 64), stored in standard MLX affine 4-bit format. Embedding / tied LM head: 4-bit. Norms and per-head attention output gates: bf16.
- File size 2.32 GB / 2.16 GiB (bf16 original: 8.22 GB).
Results (measured on an M4 Pro Mac mini, 64 GB)
| Model | Size | Mean KL vs bf16 (nats/token) | Top-1 agree | MMLU-Pro (700-question holdout) | Decode tok/s |
|---|---|---|---|---|---|
| bf16 original | 8.22 GB | 0 | 100% | 36.4% | 27.5 |
MLX 4-bit RTN (standard mlx_lm.convert -q) |
2.31 GB | 0.212 | 82.7% | 34.7% | 85.6 |
| this model (GPTQ 4-bit) | 2.32 GB | 0.051 | 89.2% | 37.7% | 85.2 |
- KL / top-1: 16 held-out sequences, 4,617 tokens.
- MMLU-Pro: a fixed 700-question subset (questions 300–999 of our 1,000-question sample) not used for any build decision; 10-choice, single run, no confidence interval reported. Accuracy is within noise of bf16, but only 82.0% of the questions bf16 answers correctly are still answered correctly (answers shift on low-confidence questions), so per-question behaviour is not identical to bf16. This is below the 90% retention bar we set internally.
- Peak memory during generation: 3.80 GB.
Usage
spark2_5 is a custom architecture not yet in mlx-lm, so this repo ships the MLX model file spark2_5.py.
pip install mlx-lm
hf download Jakevin/spark-x2.6-4b-gptq4-mlx --local-dir spark-gptq4
python spark-gptq4/run_mlx.py "Hello"
run_mlx.py registers spark2_5.py with mlx-lm and calls mlx_lm.load / generate. The original modeling_spark.py is kept for reference (PyTorch, trust_remote_code).
Limitations
- Evaluated only on MMLU-Pro subsets and KL on short text; no long-context, coding, or chat-quality evaluation.
- The holdout has 700 questions, fewer than the 1,000 we originally targeted.
- Greedy outputs of the bf16 MLX port matched the reference transformers implementation token-for-token on 5 short prompts.
License
Apache-2.0, same as the base model (see LICENSE).
Comparison chart (all MLX builds of this base model)
| Revision | GiB | KL DIV | PPL | MMLU-Pro (700) | Decode tok/s |
|---|---|---|---|---|---|
| bf16 | 7.66 | 0.0000 | 12.43 | 36.4% | 27.5 |
| MLX 4-bit RTN | 2.15 | 0.2125 | 14.23 | 34.7% | 85.6 |
| GPTQ 4-bit | 2.16 | 0.0509 | 12.51 | 37.7% | 85.2 |
| mixed48 | 2.31 | 0.0496 | 12.80 | 37.1% | 81.1 |
| tern48 (unreleased) | 2.31 | 0.1416 | 13.30 | 37.1% | 79.5 |
| mixed 3 bpw (unreleased) | 1.50 | 1.3204 | 31.22 | — | 99.4 |
| ternary 3 bpw (unreleased) | 1.50 | 1.0873 | 29.29 | — | 96.7 |
KL and PPL: 16 held-out sequences (4,601 predicted positions, full vocabulary) against the MLX bf16 model. These numbers are not comparable with KL/PPL reported by other tools (llama.cpp, exllamav3), which use different chunking and reference text.
For reference, on the same 700 MMLU-Pro questions with the same prompt and scoring, llama.cpp GGUF Q4_K_M builds scored 34.1–35.4% (XHToken official 34.1%, bartowski 35.4%) and GGUF bf16 scored 36.6%. With 700 questions the standard error of a single score is roughly ±1.8 points, so these MMLU-Pro differences are within noise; no paired significance test was run.