TL;DR: A 4.1B-parameter model in just 1.5–1.8 GB, down from the original 8.2 GB BF16 checkpoint. Small enough to make local inference practical on laptops and other memory-constrained devices.

Spark-X2.5-4B · GSQ-RCO GGUFs

Non-uniform GGUF quantizations produced with GSQ and RCO.

Independent community reproduction. These files were produced by a third party using the published GSQ and RCO methods. They are not an IST-DASLab release and carry no endorsement from the authors of either paper.

---

Overview

This repository provides GGUF quantizations of XHToken/Spark-X2.5-4B at two sizes. Uniform quantization applies one type to every weight tensor. Here each file assigns a separate type per tensor, chosen by a search that allocates precision by per-tensor sensitivity under a total size budget. The outputs are standard GGUF and run in a llama.cpp build with Spark-X2.5 support.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Read this before using these files

Perplexity: 75.7% above BF16 at 3 bits, 34.2% at 3.5 bits. MMLU-Pro (2,000 questions, paired): BF16 40.2%, 3.5-bit 31.2%, 3-bit 18.8%, chance 11.2%.

The method's authors report task-lossless results for ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, a much larger model; this run does not reach that.

---

Available files

---

MMLU-Pro

2,000 questions, fixed seed, stratified over all 14 categories. Zero-shot, lettered options, log-likelihood of A through J, no chat template. One runtime for every build: llama.cpp de25343 with CUDA on an H200. Scorer, question ids and per-question predictions ship under eval/.

Chance is 11.2%. Runtime check: the BF16 build scored on CPU gives the same 40.15%, with identical predictions on 99.0% of questions.

Per category (about 140 questions each, standard error near 4 points):

---

Diagnostics

Perplexity: 4,088 held-out tokens. IFEval: 16 prompts, strict checker, thinking off, 1,024-token cap. GSM8K: 8 questions, thinking on, 2,048-token cap, flexible extraction.

IFEval truncations: BF16 3, 3-bit 4, 3.5-bit 1 of 16. GSM8K: the 3-bit's 4/8 includes one truncated response that matched by coincidence (3/8 completed correctly); truncations 1 / 4 / 0 for BF16 / 3-bit / 3.5-bit. Eight items cannot support the 3.5-bit's 8/8.

Against same-allocation initializer controls, perplexity fell 13.3780 to 11.2842 at 3 bits and 9.9695 to 8.6206 at 3.5 bits. Paired over the eight shared chunks: t = -3.48 at 3.5 bits (p < 0.05), t = -1.35 at 3 bits (not significant).

---

Usage

llama.cpp

Requires a build with Spark-X2.5 support.

hf download pfeifferj/Spark-X2.5-4B-GSQ-RCO-GGUF Spark-X2.5-4B-GSQ-RCO-3.5bit.gguf --local-dir .

llama-server -m Spark-X2.5-4B-GSQ-RCO-3.5bit.gguf -ngl all

The 3.5-bit file passed a launch check at four slots and 131,072 total context tokens.

---

Quantization procedure

1. Candidate bank. 540 Q2_K/Q3_K/Q4_K candidates for Spark's 180 core matrices, native packed codes, no second rounding pass.
2. RCO search. Two initializations across both sizes on the shared bank; selection by lowest four-sequence calibration teacher KL, fixed before any test result.
3. Assembly. Selected per-tensor variants stitched into one GGUF.

Tied embedding/output at Q4_K; attention gates and norms at BF16/F32; native scale metadata frozen; 64 GSQ updates per stage. GSQ took 13m46s, RCO 2m13s.

---

Citation

If you use these files, please cite this release together with the base model and both methods.

This release

@misc{sparkx25gsqrco2026,
  title        = {Spark-X2.5-4B GSQ-RCO GGUF quantizations},
  author       = {Josephine Pfeiffer},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10397},
  howpublished = {\url{https://huggingface.co/pfeifferj/Spark-X2.5-4B-GSQ-RCO-GGUF}}
}

Base model

@misc{sparkx2.5,
    title  = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models},
    author = {SparkLLM Team},
    year   = {2026}
}

Methods

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

Huge kudos to the Deep Algorithms and Systems Lab (DASLab) at the Institute of Science and Technology Austria for developing GSQ and RCO and for releasing the papers and reference implementations publicly.

Thanks also to ella for giving me the idea and supporting me at every step of the process, and Chris van Hoof (Red Hat) for contributing the compute on which these quantizations were produced and evaluated.

---

License

These quantized weights inherit the license of the base model, XHToken/Spark-X2.5-4B (Apache-2.0); the original license text ships as LICENSE. The GSQ and RCO tooling is released by DASLab under its own repository licenses.