Q4_K I/O) GGUFThis repository provides a 3.494 bits-per-weight (3.64 GiB / 3,911,259,680 bytes) mixed-precision GGUF quantization of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (revision 2367e865d009c13ac81713a2878291d33ab28177), built using Global Sensitivity Quantization (GSQ) (arXiv:2604.18556) and Riemannian Cost-Constrained Optimization (RCO) (arXiv:2605.00649).
The goal of this quantization is to achieve a 4.58× size reduction from the 16-bit BF16 checkpoint (16.68 GiB) while preserving the model's agentic coding intelligence, succinct <think> reasoning, and multi-turn <tool_call> accuracy for long-context local inference on unified-memory devices.
| File | Quantization | Size | Effective bpw |
SHA-256 |
|---|---|---|---|---|
MiMo-V2.6-9B-gsq-rco-3p5-q4io.gguf |
GSQ + RCO (3.5 bpw body + Q4_K I/O embeddings) |
3.64 GiB (3,911,259,680 B) |
3.494 |
3d6004ad3be2aa0e0b7b9ae1add40b92a62e67f8c8d89fcdcabd1987fd0e00e8 |
MiMo-V2.6-9B-gsq-rco-3p5-q4io.json |
Per-tensor GGUF qtype assignment & byte receipt |
6.7 KB |
— | — |
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B uses the 32-layer qwen3_5_text hybrid architecture (24 linear Gated DeltaNet blocks + 8 full Gated Attention blocks + dense SwiGLU MLP, totaling 192 block projection matrices and two untied 248,320 × 4,096 vocabulary matrices token_embd.weight and output.weight).
imatrix Candidate Bank:
Each of the 192 block matrices is initialized across 6 GGUF formats (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_XS, IQ4_XS / Q4_K) using llama-quantize --imatrix.IQ2_S, IQ3_XXS, IQ3_XS, IQ4_XS — 768 total tensor candidates) are decomposed into their native GGUF codebook grid indices and continuous super-block/sub-block scale factors. Discrete grid codes are optimized via Gumbel-Softmax temperature annealing jointly with continuous scale factors (AdamW, 80 steps) against cached BF16 teacher input activations to minimize relative output MSE:
$$\mathcal{L}{\text{GSQ}}(W_q) = \frac{|X W_q^\top - X W{\text{BF16}}^\top|F^2}{|X W{\text{BF16}}^\top|_F^2}$$
Refined tensors are kept only when their hard-packed GGUF bytes strictly improve held-out selection NMSE (535 / 768 candidates improved over llama-quantize --imatrix) and verify byte-for-byte through libggml's C decoder.3,917,288,928 bytes). Each step projects Gumbel-perturbed scores onto a feasible integer assignment via a multiple-choice knapsack solver (16 KiB byte quanta) and differentiates straight-through hard decoded weights against full-model top-64 + tail BF16 teacher KL divergence on assistant tokens.Q4_K Untied Vocabulary Embeddings (q4io):
Because token_embd.weight and output.weight are untied [248320, 4096] matrices (2.034B parameters = 22.7% of the model), pinning both to Q4_K (4.5 bpw) while allocating the remaining byte budget across the GSQ-RCO refined transformer body reduces full-vocabulary teacher-student KL divergence by 17.0% (0.08885 → 0.07377).Calibration, selection, and held-out test splits use 2,048-token windows (64 train / 8 selection / 16 held-out test sequences) anchored across </think> boundaries and masked exclusively to assistant response tokens (<think>...</think>, code, and <tool_call>...</tool_call>):
| Domain | Dataset | Split (Train / Sel / Test) |
RCO KL Weight |
|---|---|---|---|
coding |
theblackcat102/evol-codealpaca-v1 |
18 / 2 / 4 |
2.0 |
code_reasoning |
open-r1/Mixture-of-Thoughts (code) |
16 / 2 / 4 |
2.0 |
tools |
nvidia/Nemotron-Agentic-v1 (tool_calling) |
16 / 2 / 4 |
2.0 |
reasoning |
open-r1/Mixture-of-Thoughts (math) |
10 / 1 / 2 |
1.0 |
general |
Salesforce/wikitext (wikitext-103-raw-v1) |
4 / 1 / 2 |
0.5 |
Full-Vocabulary KL & NLL vs. Unquantized BF16)Evaluated over the 16 sealed held-out test sequences (32,768 tokens, unweighted full-vocabulary KL divergence and negative log-likelihood against the unquantized BF16 teacher):
| Model | Size (GiB) |
Effective bpw |
Full-Vocab Teacher KL ↓ | Student NLL ↓ (BF16 = 0.9583) |
coding KL ↓ |
code_reasoning KL ↓ |
tools KL ↓ |
|---|---|---|---|---|---|---|---|
BF16 (XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B) |
16.68 GiB |
16.000 |
0.00000 |
0.95826 |
0.00000 |
0.00000 |
0.00000 |
MiMo-V2.6-9B-gsq-rco-3p5-q4io.gguf (Ours) |
3.64 GiB |
3.494 |
0.07377 |
0.98866 |
0.04118 |
0.04277 |
0.06465 |
IQ3_XS (llama-quantize --imatrix) |
3.55 GiB |
3.405 |
0.08754 |
0.99640 |
0.04579 |
0.04590 |
0.07727 |
MiMo-V2.6-9B-gsq-rco-3p5.gguf (IQ3_XXS I/O) |
3.65 GiB |
3.500 |
0.08885 |
0.99749 |
0.04687 |
0.04606 |
0.07157 |
MiMo-V2.6-9B-gsq-rco-3p3.gguf |
3.44 GiB |
3.300 |
0.10588 |
1.00580 |
0.05485 |
0.05743 |
0.08976 |
IQ3_XXS (llama-quantize --imatrix) |
3.32 GiB |
3.184 |
0.10726 |
1.00638 |
0.05626 |
0.05686 |
0.09702 |
MiMo-V2.6-9B-gsq-rco-3p1.gguf |
3.24 GiB |
3.112 |
0.12358 |
1.01577 |
0.06306 |
0.06808 |
0.10862 |
llama.cpp Usage & Sampling ParametersFollowing Xiaomi's official MiMo-V2.6-Distill-Qwen-9B generation_config.json:
temperature: 0.6top_p: 0.95top_k: 20min_p: 0.0repeat_penalty: 1.0llama-server \
-m MiMo-V2.6-9B-gsq-rco-3p5-q4io.gguf \
-ngl all -fa on -c 32768 -ctk q8_0 -ctv q8_0 \
--jinja --reasoning on --reasoning-format deepseek \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B