Mistral-Small-4-119B-2603 · GSQ-RCO IQ3_S GGUF

A personal quantization, made on end-user hardware. One person built this file on one home desktop (AMD Ryzen 9 5900X, 64 GB of RAM, an RTX 3090 and an RTX 3070 Ti, a 4 TB NVMe SSD), not on a lab cluster. It is not an official release of IST Austria or Mistral AI: it applies their published work (see Credits).

A mixed-precision GGUF of mistralai/Mistral-Small-4-119B-2603 (a 119B mixture-of-experts model, 6.5B active parameters per token, taking text and images, with instruct and reasoning modes), quantized with GSQ and RCO, the two methods of IST Austria's DAS Lab, following the procedure of their GSQ-RCO GGUF releases:

• GSQ (Gumbel-Softmax Quantization) refines the integer codes of every tensor inside a GGUF type, against the inputs that tensor actually sees, at every candidate type;
• RCO (Riemannian Constrained Optimization) then picks one type per tensor, under an exact size budget, by minimizing the KL divergence of the whole model's output from the BF16 model's.

On top of that procedure, an end-to-end tuning adjusted the values GSQ and RCO leave continuous (the block scales of the quantized tensors, the norms and the routers) against the BF16 model's output, with every integer code and every type kept: see the Changelog.

The budget is the one ISTA uses for their IQ3_S releases: 3.50 bits per weight over the whole file. The result is not a uniform IQ3_S: each tensor carries the type RCO assigned to it, from IQ2_XS to Q6_K.

Changelog

• 2026-11-05: the small tensors at Q6_K, retuned. RCO's search had left the types of most attention, shared-expert and embedding tensors (2.5 % of the weights) nearly undecided, and 19 of them at 2 to 3.4 bits; they are now all Q6_K, from the same GSQ candidates, and the 0.2 GB that costs is paid by two expert tensors RCO was as unsure about, one type down each (step 8 of How it was made). The end-to-end tuning was then run again from the start on the new file. The file names stay; the file is 12 MB smaller than before. Mean KL divergence from BF16, every file measured the same way:

  Domain	First release	2026-11-04	Now	Change
  WikiText-2 test	0.1983	0.1696	0.1579	−6.9 %
  C++ source code	0.0709	0.0623	0.0570	−8.5 %
  Open-Platypus, held-out rows	0.1178	0.0964	0.0887	−8.0 %

  The 2026-11-04 file stays available at revision 957d11e9c5a9d50e37d835f2724c79e30b588f87, the first release at 183472c837120d277b67a08b8ce630f1b7035042.

• 2026-11-04: end-to-end tuned weights. The block scales of the quantized tensors, the norms and the routers were tuned against the BF16 model (step 7 of How it was made). Every tensor keeps its type: the size, the file names and the way to load them do not change, only the values do. Mean KL divergence from BF16, with the first release measured again on the same build, settings and GPU as the new file (the first card listed 0.1984, 0.0709 and 0.1170, from a separate run):

  Domain	First release	Now	Change
  WikiText-2 test	0.1983	0.1696	−14.5 %
  C++ source code	0.0709	0.0623	−12.2 %
  Open-Platypus, held-out rows	0.1178	0.0964	−18.2 %

  The first release stays available at revision 183472c837120d277b67a08b8ce630f1b7035042, for example with hf download fajardc/Mistral-Small-4-119B-2603-GSQ-RCO-GGUF --revision 183472c837120d277b67a08b8ce630f1b7035042.

• 2026-11-03: first release.

Files

This repository holds the language model only, split in two parts under Hugging Face's file-size limit. Load the first part; llama.cpp finds the second one next to it.

• Vision. Mistral-Small-4 also takes images, which needs a vision projector (mmproj) GGUF loaded next to the model. The projector is not part of this quantization, so any projector of Mistral-Small-4-119B-2603 works, for example bartowski's mmproj-mistralai_Mistral-Small-4-119B-2603-bf16.gguf, or one converted from the original checkpoint with llama.cpp's convert_hf_to_gguf.py --mmproj.
• Speculative decoding. The model has no multi-token-prediction (MTP) head. Mistral publishes a separate EAGLE draft head, mistralai/Mistral-Small-4-119B-2603-eagle, for engines that support it; llama.cpp does not load it at the time of writing.
• Chat template. The template embedded in the file is Mistral's, with one fix: its tool-result branch now passes render_content the support_thinking argument it leaves out. Python's Jinja fills the missing argument in, llama.cpp's template engine does not: with the original template, every request that carries a tool result fails with "Not enough arguments provided to 'render_content'".

Quality

Mean KL divergence and same-top-1 rate of the next-token distribution against the BF16 GGUF of the same model, 100 chunks of 512 tokens per domain, on three domains held out from every calibration and tuning set:

The perplexity ratio has been below 1 since the end-to-end tuning, which does not make the file better than BF16: on average it gives the correct token slightly less probability than BF16 (1.7, 0.5 and 1.1 points less on the three domains, against 1.4, 0.3 and 0.7 before the tuning), but more where BF16 gives it little, which is what perplexity weighs most. The KL divergence, which compares the whole distribution, is the measure of how close the file is to BF16.

What RCO chose

The dense tensors are all Q6_K since the 2026-11-05 release: the attention projections (attn_q_a, attn_q_b, attn_kv_a_mqa, attn_v_b, attn_output), the shared expert, token_embd and output; norms and routers are F32. RCO's own choice for them, in the first two releases, was Q6_K for the most part with some Q5_K, Q4_K, IQ4_XS and a few lower, and token_embd at Q5_K (see step 8). attn_k_b is Q8_0 and outside the search: its rows are 64 values wide, which no K-quant or IQ type can hold. The routed experts, which are most of the file, get the low types, unevenly across depth:

Tensor counts over the file: Q6_K 212, IQ3_XXS 47, IQ3_S 40, IQ4_XS 38, Q8_0 36, Q4_K 26, Q5_K 17, IQ2_S 12, IQ2_XS 6, F32 181. The type of every tensor can be read from the file itself, for example with gguf-py's gguf-dump.

How it was made

1. Conversion. The Hugging Face checkpoint, in FP8 with per-tensor and per-expert scales, was converted to a BF16 GGUF with llama.cpp's convert_hf_to_gguf.py, each weight multiplied by its scale. The experts' gate and up projections, which the converter merges into one tensor, were stored as two tensors (the same values; llama.cpp loads either layout).

2. Candidates. Every tensor was quantized at each candidate type (IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q4_K, Q5_K, Q6_K) with llama.cpp's llama-quantize and bartowski's importance matrix of this model (822 chunks of 512 tokens, every expert of every layer covered).

3. Hessians. The BF16 model was run on 192 sequences of 2,048 tokens of the calibration mix (below), one decoder layer at a time, collecting the input Hessian H = XᵀX / n of every linear layer (for the MLA projections that read a latent, attn_q_b and attn_v_b, the Hessian of that latent), and a separate Hessian per routed expert from the tokens routed to it.

4. GSQ. Each candidate tensor was refined inside its GGUF type: the integer codes (for the K-quant types, a shift of −2…+2 per weight; for the IQ codebook types, the grid entries nearest each 4- or 8-value group) are chosen by a Gumbel-softmax relaxation that minimizes the layer's output error ‖(W̃ − W)·L‖²_F with H = L·Lᵀ, while the block scales and signs stay as llama.cpp set them (200 steps, Lion, lr 6e-4). A refined tensor is kept only if its hard-rounded error went down. Experts are refined each against its own Hessian, shrunk toward the layer's pooled Hessian in proportion to how few tokens reached it; an expert routed fewer than 64 calibration tokens keeps llama.cpp's codes. Mean reduction of the layer error against llama.cpp's own imatrix quantization:

   IQ2_XS	IQ2_S	IQ3_XXS	IQ3_S	IQ4_XS	Q4_K	Q5_K	Q6_K
   28.7 %	28.9 %	33.0 %	39.0 %	36.5 %	30.9 %	33.4 %	36.8 %

5. RCO. One type per tensor group (326 groups; the gate and up projections of a layer's experts share one) was searched under the 3.50 bpw whole-file budget, minimizing KL(BF16 ‖ quantized) over the full vocabulary on 256 calibration sequences of 2,048 tokens, with RCO's own optimizer code (Gumbel straight-through samples under the budget, Riemannian Adam on the budget manifold, its sensitivity initialization and final knapsack rounding; 200 steps, 4 samples per step, temperature 1.0 → 0.01, lr 0.1). The model, too large for memory, was streamed one decoder layer at a time, and the gradient of the choice was formed from each layer's inputs and output gradients over the GSQ candidates.

6. Assembly. The chosen GSQ candidates were copied into one file as they are.

7. End-to-end tuning. The values the file keeps continuous were then tuned through the whole quantized model, minimizing KL(BF16 ‖ quantized) of the next-token distributions: one factor per block of 256 weights on its 16-bit scale (for Q4_K and Q5_K on its minimum too, so the whole block scales), in every quantized tensor but the token embedding and the Q8_0 attn_k_b (462 million factors), and in the F32 tensors a factor on each weight of every RMSNorm and a shift on each weight of the 36 expert routers. The integer codes and every type stay as GSQ and RCO set them: only continuous values move, as in the P step of PV-Tuning. Adam on batches of two sequences of 2,048 tokens, the model streamed one decoder layer at a time as for RCO: 150 steps on the block scales alone, against the full-vocabulary distributions of RCO's calibration sequences (learning rate 5e-4), then 200 steps on block scales, norms and routers together, against the top-128 distributions (and the rest of the mass) of 2,400 sequences of a larger mix of the same sources without the papers, the learning rate decaying from 5e-4 to zero along a cosine.

8. The small tensors at Q6_K (2026-11-05). RCO's search had ended with 104 of its 326 groups decided; 135 others had their winning type under 75 % probability, nearly all of them attention, shared-expert and embedding tensors, with runner-ups three or four bits away: for tensors that small (2.5 % of the weights) the bit cost barely moves the objective, so the search never resolved them, and 19 of them sat at 2 to 3.4 bits. Setting every one of them to its GSQ candidate at Q6_K costs 0.2 GB and, before any tuning, lowers the KL divergence by 7.2, 8.4 and 7.2 % on the three domains. The 0.2 GB was paid inside the budget by two expert tensors whose choice RCO had been as unsure about, each one type down (layer 8's down projection Q6_K → Q5_K, layer 11's gate and up projections Q4_K → IQ4_XS), at no measurable cost. Step 7 was then run again from the start on the new file, with the same schedule.

ISTA publishes the methods and their code, but neither GGUF tooling for GSQ nor an allocation for this architecture: here GSQ was reimplemented for GGUF types after the paper, and RCO was run through its own code. The end-to-end tuning of step 7 is not part of ISTA's procedure.

Hardware and time

Everything ran on the desktop named above. The BF16 weights (about 238 GB) are almost four times its RAM, so the Hessian collection, GSQ, RCO and the end-to-end tuning all stream the model one decoder layer at a time, and the candidate files (the eight types, then their GSQ refinements, about 0.9 TB together) live on the SSD. GSQ shared its work between the two GPUs and took about 39 hours; RCO ran on the RTX 3090 alone, about 9.5 hours for its 200 steps, and so did the end-to-end tuning, about 4 hours for its 350 steps and the BF16 targets; conversion, candidates and Hessians took a few hours more. The second allocation's tuning took another 4 hours.

Calibration mix

Documents of five domains interleaved, so that any prefix of the text is balanced, in the spirit of GSQ's calibration mixture: WikiText-2 train, Python source code, OpenThoughts-114k (problems, reasoning traces and solutions), Open-Platypus (instruction/response pairs, the first 2,000 rows held out for evaluation), and the text of the GSQ and RCO papers.

Usage

Any llama.cpp build recent enough for mistral4 loads it. For example, with the expert weights kept in system RAM and everything else on the GPU:

llama-server -m Mistral-Small-4-119B-2603-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
  --mmproj mmproj-mistralai_Mistral-Small-4-119B-2603-bf16.gguf \
  -c 262144 -ngl 99 --cpu-moe --jinja --temp 0.7

--mmproj is only needed for image input. This command was checked on upstream llama.cpp (build 11669, commit 620d2225c) with text, image input, tool calls and their results, and reasoning, on every release; MLA keeps the cache of the full 262,144-token context small. --n-cpu-moe N keeps only the first N layers' experts in RAM, for GPUs with room for more.

Mistral-Small-4 answers directly by default and reasons step by step on request: pass "chat_template_kwargs": {"reasoning_effort": "high"} in a request (or --chat-template-kwargs '{"reasoning_effort": "high"}' to the server for every request); "none" is the default, and the template accepts no other value. llama.cpp returns the reasoning in reasoning_content. Mistral recommends temperature 0.7 with reasoning, and between 0.0 and 0.7 without, depending on the task.

Credits

• Mistral AI for Mistral-Small-4-119B-2603, the model quantized here, and its chat template.
• IST Austria, DAS Lab for GSQ and RCO (GSQ code, RCO code) and for the GSQ-RCO GGUF procedure and size budget this file follows.
• llama.cpp / ggml for the GGUF format, its quantization types, conversion and quantization tools, and the runtime used for evaluation.
• bartowski for the importance matrix of this model and its vision projector GGUF.
• The authors of WikiText-2, OpenThoughts-114k and Open-Platypus, used for calibration and evaluation.

Citation

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

Apache 2.0, as the base model.