Spark-X2.5-4B-MLX-8bit

An 8-bit MLX conversion of Spark-X2.5-4B, a compact agentic model from the SparkLLM team with a native 1M-token context window. This is the maximum-fidelity build: perplexity is statistically identical to the bf16 original, at 4.1 GB on disk, ~4.1 GB GPU memory at runtime — half the original's footprint.

Same recipe as the 4-bit and 6-bit builds: everything quantized including the tied embedding table, with the head-wise attention output gates (g_proj, ~187K params) kept in BF16 per the upstream guidance. The upstream team recommends 8-bit or BF16 weights for tool calling, which makes this the build to pick for agent harnesses.

Quantization details


Format	MLX affine, 8-bit, group size 64
Effective bits per weight	8.503
Model size	4.1 GB (from 8.2 GB bf16)
Kept at BF16	`self_attn.g_proj` gates + RMSNorm weights
Embeddings	8-bit (tied lm_head)

Converted with mlx-lm 0.31.3 and the official Spark-MLX-LLM architecture support, orchestrated by Hermes Agent.

Loading

spark2_5 isn't in the latest mlx-lm PyPI release yet (it has landed on mlx-lm's GitHub main), so this repo ships the architecture file (spark2_5.py) and loads it via mlx-lm's model_file mechanism. Requires mlx-lm ≥ 0.31. No extra packages needed:

pip install -U mlx-lm

python3 -m mlx_lm.generate \
  --model hermitdave/Spark-X2.5-4B-MLX-8bit \
  --prompt "Explain why hybrid attention saves memory at long context." \
  --max-tokens 512

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/Spark-X2.5-4B-MLX-8bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the capital of Anhui Province?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Alternatively, install Spark-MLX-LLM for their spark-mlx-* CLI wrappers and tool-call parsing.

Benchmarks (measured, not copied)

Run on an M3 Max (64 GB) by Hermes Agent at conversion time. Perplexity on ~24.6k tokens of Pride and Prejudice (Gutenberg), 1024-token windows:

	bf16 base	this 8-bit	6-bit	4-bit
Perplexity ↓	14.45 ± 0.27	14.49 ± 0.27	14.56 ± 0.27	16.33 ± 0.31
Disk size	8.2 GB	4.1 GB	3.1 GB	2.2 GB
Runtime GPU memory	~8.3 GB	4.10 GB	3.15 GB	2.26 GB
Decode speed	—	20 tok/s	20 tok/s	21 tok/s

The 8-bit perplexity is within a rounding error of bf16 — quantization loss is unmeasurable at this precision. The quality cliff in this family sits between 6-bit and 4-bit (+13% perplexity at 4-bit); 8-bit and 6-bit are both statistically clean.

Quality smoke tests all passed at temp 0: 53 × 42 = 2226 (matches the vendor's reference output), train speed 80 km/h, capital of Anhui → Hefei, correct Chinese-language arithmetic, and a well-formed get_weather(city="Paris", unit="celsius") tool call.

Architecture

Spark-X2.5 uses a hybrid attention pattern — 3 sliding-window layers followed by 1 full-attention layer, repeating:

36 layers (27 sliding + 9 full), window 512
Hidden 2560, 16 attention heads / 4 KV heads, head dim 256
GQA + partial RoPE on full-attention layers (25% rotary, θ=5M); full RoPE θ=10k on sliding layers
Head-wise sigmoid attention output gates
Parallel GELU MLP, intermediate 10240
Vocab 131,072, tied embeddings
Native context: 1,048,576 tokens

Thinking mode is on by default via the chat template; pass enable_thinking=False to apply_chat_template to disable. Recommended sampling: temperature 1.0, top_p 0.95.

License

Apache 2.0, same as upstream. The bundled spark2_5.py is derived from Spark-MLX-LLM (Apache 2.0).

Attribution

Upstream model: XHToken/Spark-X2.5-4B by the SparkLLM team
Architecture support: Spark-MLX-LLM
Conversion & validation: Hermes Agent by Nous Research, using mlx-lm 0.31.3

Citation

@misc{sparkx2.5,
    title  = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models},
    author = {SparkLLM Team},
    year   = {2026}
}

hermitdave/Spark-X2.5-4B-MLX-8bit