Spark-X2.6-1.7B-MLX-slq-4.5bpw
SLQ (Statistically-Lossless Quantization) mixed-precision MLX build of XHToken/Spark-X2.6-1.7B, quantized with the slq-mlx port.
Packed artifact: 1.296 GiB on disk · 4.50 bpw average · 168 layers packed.
Quantization details
| Method | SLQ (per-layer greedy heuristic) |
| Group size | 128 |
| Effective bits/weight | 4.4989 |
| Disk size | 1.296 GiB (BF16 base: 3.205 GiB → 2.47× smaller) |
| Runtime memory | ~1.4 GiB |
| Bitwidth distribution | 8-bit: 16.7% (all g_proj), 5-bit: 51.8%, 4-bit: 31.5% |
| Scale/zero dtype | float16 with automatic float32 fallback |
Scale dtype selection
The packer defaults to float16 for quantization scales and zero-points. This halves the dequantization bandwidth and is safe for models whose per-group scales/zeros lie within the float16 range (±65,504).
Automatic fallback: if any layer’s scale or zero exceeds the float16 safe limit, the packer logs a warning and writes that layer’s scales/zeros as float32. This prevents silent overflow to inf or underflow to 0.
For this Spark model, all 168 layers safely used float16 (max scale: 0.18, max zero: −1.61).
Throughput (measured)
Prompt: "Explain why hybrid attention saves memory at long context in exactly 80 words."
Max new tokens: 128 · Temperature: 0.0 · Top-p: 0.95
Hardware: Apple M3 Max (64 GB) · MLX 0.32.4
| Model | Tokens generated | Time | Throughput |
|---|---|---|---|
| BF16 base | 127 | 1.57 s | 81.0 tok/s |
| SLQ 4.50 bpw (float16 scales) | 128 | 0.81 s | 157.5 tok/s |
Float16 scales/zeros recovered the dequantization bandwidth and made the packed model ~1.9× faster than BF16 on this hardware. The fused QKV/FFN kernels from spark-mlx-ext were not installed, so this is the unfused-path throughput.
Loading
pip install -U mlx-lm
python3 -m mlx_lm.generate \
--model ~/Projects/ml/models/Spark-X2.6-1.7B-MLX-slq-4.5bpw \
--prompt "What is 3 times 7?" \
--max-tokens 64
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
model, tokenizer = load("~/Projects/ml/models/Spark-X2.6-1.7B-MLX-slq-4.5bpw")
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is 3 times 7?"}],
tokenize=False,
add_generation_prompt=True,
)
sampler = make_sampler(temp=0.0, top_p=0.95)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, sampler=sampler))
Validation (measured)
Perplexity on FineWeb-Edu (32 × 1024 tokens):
| Model | PPL | Ratio |
|---|---|---|
| BF16 base | 1.4836 | 1.000 |
| SLQ 4.50 bpw | 1.5609 | 1.052 |
Task-level zero-shot recovery (100 examples/task, paired):
| Task | BF16 | SLQ | Recovery | 95% CI |
|---|---|---|---|---|
| arc_challenge | 0.3300 | 0.3300 | 1.0000 | [−0.0480, +0.0480] |
| arc_easy | 0.5200 | 0.5100 | 0.9808 | [−0.0859, +0.0659] |
| hellaswag | 0.3400 | 0.3600 | 1.0588 | [−0.0192, +0.0592] |
| winogrande | 0.5000 | 0.4200 | 0.8400 | [−0.1677, +0.0077] |
| mean | 0.9699 |
Verdict: not task-lossless (mean recovery 0.97 < 0.99 paper threshold), consistent with the per-layer greedy allocator at 4.5 bpw. PPL increase of ~5.2% matches the paper's reported Task-Lossless regime.
Architecture
Spark-X2.6-1.7B uses the same hybrid attention pattern as Spark-X2.5-1.7B:
- 28 layers (21 sliding-window + 7 full-attention), window 512
- Hidden 2048, 8 attention heads / 2 KV heads, head dim 256
- Fused QKV projection (
self_attn.q_k_v_proj) - Head-wise sigmoid attention output gates (
self_attn.g_proj) - Parallel GELU MLP, intermediate 6656
- Vocab 131,072, tied embeddings
Attribution
- Upstream model: XHToken/Spark-X2.6-1.7B by the SparkLLM team
- Architecture support: Spark-MLX-LLM
- Quantization: slq-mlx (IST-DASLab SLQ port to MLX)
- Conversion & validation: Hermes Agent by Nous Research
License
Apache 2.0, same as upstream.