Spark-X2.6-4B-abliterated · FIT-GGUF
Five fidelity tiers of a 4B hybrid-attention model — each one the smallest GGUF that provably meets its KL gate, re-measured on its own shipped bytes.
2.18 GiB · MINIthe verified minimum at each fidelity7.66 GiB · BF16 🧭 About FIT-GGUF — the tool behind these filesEvery FIT tier here was planned, executed and verified by FIT-GGUF, an open-source, deterministic tensor-level planning layer on top of llama.cpp quantization. Standard GGUF quantization asks you to pick one of a handful of presets; FIT-GGUF instead asks what quality do you want, then finds and verifies the smallest GGUF that demonstrably meets it.
Traditional GGUF gives you presets. FIT gives you a fidelity contract: macro KL ≤ tier anchor, measured against this model's own BF16 on five fixed domains.
| Deterministic size prediction & byte-exact delivery | ✅ Validated (G2 gate, delta = 0) |
| Universally optimal tensor allocation | ⚠️ Not established — FIT claims verified fidelity contracts, not a universal quality optimum |
Method, pre-registered research record and the fit CLI are all open source: github.com/Scorp1o117/FIT-GGUF
The source model is an abliterated, refusal-removed model with no meaningful built-in guardrails, and may comply with harmful, illegal or unsafe requests. Use it only where you can provide appropriate moderation, access control and legal review. Do not deploy it to end users without your own safety layer.
源模型经过拒答方向消融，不具备可靠的内置安全护栏；请仅在合法、受控、具备审核与滥用防护的环境中使用，使用者自行承担部署责任。
🧬 Abliteration — input-gated, documentedThe refusal direction was ablated locally with the abliterix pipeline using input-gated ablation. A classic rank-1 edit is ΔW = −w · v (vᵀW) — the same output direction v for every input, harmless ones included, which is where the KL bill comes from. The gated form replaces the input-side vector with a per-component direction g chosen to maximise refusal suppression per unit of perturbation, g ∝ S_h⁻¹ μ_r: the harmful-prompt mean whitened by the harmless second-moment matrix, so the edit lands where harmless traffic does not live.
| Metric | Delivered |
|---|---|
| Refusals (100 held-out harmful prompts) | 7 (99 before ablation) |
| Abliteration perturbation, 3-token full-distribution KL | 0.0477 |
| Previous best on this model (same refusal bar, no gate) | 0.2134 at 10/100 refusals — the gated direction cut it by 78% while refusing less |
Two different KLs — do not mix them. The 0.0477 above is the abliteration's perturbation of the full-precision model. Every other KL number on this page is quantization divergence against this repository's BF16 file. They are measured on different objects and are not comparable.
A structured 150-trial search per arm, with a matched control, is what produced the gated direction: the ungated arm could not reach a 10/100 refusal bar below 0.17 KL, while the gated arm reached 7/100 at 0.0477 — the two refusal bands do not overlap.
📦 Everything in this repositoryFive FIT tiers and the BF16 reference — every file measured by the same evaluator, on the same five slices, against the same reference logits. The last column is the honest claim: the smallest native llama.cpp preset that would clear the same gate, so you can see what FIT bought.
| Family | File | GiB | Macro KL ↓ | Same-top ↑ | Gate | vs native |
|---|---|---|---|---|---|---|
| FIT REFERENCE | …-FIT-REFERENCE-3G-Q6_K.gguf | 3.147 | 0.01813 | 95.01% | KL ≤ 0.02 | native Q6_K is the floor |
| FIT QUALITY | …-FIT-QUALITY-3G-Q5_K_M.gguf | 2.796 | 0.04940 | 91.62% | KL ≤ 0.05 | −11.2% |
| FIT BALANCED | …-FIT-BALANCED-3G-Q4_K_M.gguf | 2.511 | 0.09172 | 87.80% | KL ≤ 0.10 | −6.3% |
| FIT COMPACT | …-FIT-COMPACT-2G-IQ4_XS.gguf | 2.213 | 0.13950 | 84.97% | KL ≤ 0.15 | native IQ4_XS is the floor |
| FIT MINI | …-FIT-MINI-2G-Q3_K_M.gguf | 2.177 | 0.19670 | 82.45% | KL ≤ 0.20 | −3.7% |
| BF16 | …-BF16.gguf | 7.662 | 0 | 100% | The reference every number above is measured against | — |
| imatrix | …-BF16-imatrix.gguf | 0.007 | — | — | 500×512 chunks, 216 entries — re-quantize this model yourself | — |
The five FIT tiers are 12.84 GiB where the smallest passing native preset in each would be 13.45 GiB — 0.60 GiB (4.5%) smaller.
Two of the five tiers are the native preset, and that is the honest answer for this model. Inside the bracket that separates a failing preset from a passing one, the planner's reachable artifact sizes are ~28 MB apart, and no reachable size exists in between — ten fresh evaluations per tier confirmed it. The other three tiers are genuine tensor-level FIT recipes, placed below and to the left of the native ladder.
📈 Measured qualityllama.cpp b1-a37cc46 · c=512 b=512 · five domains · vs the BF16 in this repo · 中文大图
The native ladder is the reference curve; the FIT tiers sit below and to the left of it wherever a gap exists. The native ladder points themselves:
| IQ2_XXS 1.30G 4.115 | IQ2_XS 1.39G 2.484 | IQ2_M 1.52G 0.920 | Q2_K 1.67G 1.169 | IQ3_XXS 1.68G 0.498 | IQ3_XS 1.78G 0.490 | IQ3_M 1.90G 0.405 | Q3_K_M 2.02G 0.312 | IQ4_XS 2.21G 0.140 | Q4_K_S 2.26G 0.152 | Q4_K_M 2.42G 0.133 | Q5_K_S 2.68G 0.058 | Q5_K_M 2.77G 0.061 | Q6_K 3.15G 0.018 | Q8_0 4.07G 0.004 |
Read the curve, not just the tiers. The gates are deliberately capped at KL 0.02–0.20, so the big native presets are cleaner than every tier — Q6_K at 3.15 GiB has KL 0.0181, cleaner than the REFERENCE gate. If you want maximum fidelity rather than minimum size, take Q6_K or the BF16 file.
🚀 Run itThis is the standard spark2_5 architecture. It needs a recent upstream llama.cpp (conversion/spark2_5.py landed in PR #27868, merged 2026-09-06) — no custom fork, but not an ancient build either. The measurements here used b1-a37cc46.
Text only
./llama-server -m Spark-X2.6-4B-abliterated-FIT-BALANCED-3G-Q4_K_M.gguf -ngl 99 -c 32768
./llama-cli -m Spark-X2.6-4B-abliterated-FIT-MINI-2G-Q3_K_M.gguf -ngl 99 -c 8192 -n 256 -p "Hello"
The chat template is embedded in every file, so LM Studio, KoboldCpp, Jan and friends load them without extra configuration. Spark2.6 enables its thinking block by default — pass enable_thinking=false through your chat template if you want the answer without the reasoning trace.
On context length: the upstream card and config.json declare max_position_embeddings = 1048576, and the card shows a server configured for a maximum context of 1,048,576 tokens — this repository did not measure it. It is reachable in principle because only 9 of the 36 layers are full attention (RoPE θ 5,000,000, partial RoPE 0.25) while the other 27 use a fixed 512-token sliding window, so the KV cache grows at roughly a quarter of a pure full-attention model. Start at -c 32768 and work up: on a 32 GiB card the MINI tier leaves real room for a long context, and the REFERENCE tier is mostly there to be the fidelity ceiling.
| Runtime | llama.cpp b1-a37cc46 (master) · Windows 11 · CUDA 13.4 · RTX 4080 SUPER (sm_89) |
| Command shape | llama-perplexity -ngl 99 -t 16 -c 512 -b 512 --kl-divergence … |
| Reference | This model's own BF16 logits (the BF16 GGUF shipped here) |
| Domains | wiki_test · wiki_valid · Chinese · code · agent_chat (five fixed 64 KiB slices, macro mean) |
| Calibration | 15-point standard preset ladder + gap probes; tier search by size bisection (16 MiB bracket tolerance); guard profile scope exact_model |
| Weights binding | source_weights_sha256 = 9c2abf3d50e80352… — the registry entry is keyed to these exact weights |
Tier verification: a FIT tier ships only if macro KL ≤ its anchor. Each shipped file was re-evaluated on its own bytes after quantization and reproduced its search-time KL exactly (G2 delta +0 on all five — the delivered byte count equals the re-finalized prediction to the byte).
Allocator scope: the stock balanced policy with precision floors applied where the window's candidate set cannot reach a tensor, using this model's own importance matrix. No model-specific refine profile. What is claimed is deterministic size planning plus measured verification of these specific artifacts — not a universally optimal allocation.
The importance matrix was rebuilt, not inherited. The original APEX calibration corpus no longer exists, so it was reconstructed from public sources (wikitext-2 train, the eaddario imatrix-calibration corpora, and MiMo-V2.6-RL-oss prompts), interleaved so that any prefix of the file is domain-balanced, and the five frozen evaluation slices were excised from the calibration text so the imatrix never saw the measured text. Five candidate corpora were compared end-to-end at a fixed preset; the spread in macro KL was 13.9% and the selected one matched the best within 2%.
Sizes in the file names are measured bytes, not budgets. The tool refuses to emit an artifact whose size it cannot predict to the byte.
🧩 Included — and not included✅ 5 gate-verified FIT tiers · ✅ BF16 reference GGUF (the exact weights every measurement above is taken against) · ✅ the importance matrix used for every quantization · ✅ chat template embedded in every file · ✅ checksums (SHA256SUMS.txt) · ✅ labelled quality curve and the raw per-domain JSON (results/)
❌ no 2-bit-and-below classes — measured collapse on this model (macro KL 0.92–4.11), excluded by design · ❌ no MTP head · ❌ no GGUF above Q6_K for the text weights; take the BF16 file if you need the exact weights
The abliteration was performed locally from XHToken/Spark-X2.6-4B; this repository contributes the abliterated weights' quantization plans, the artifacts and the measurements.
🔍 Verify & reproducesha256sum -c SHA256SUMS.txt
The evaluation slices are public in the FIT-GGUF repository; the calibration ladder, gap probes, search audit, the per-tier recipes and the guard profile for this release are retained in the FIT-GGUF experiment record. The five-domain .kld reference logits are not stored there — they regenerate from the BF16 GGUF shipped here and are checked against reference-manifest.json, which pins each domain's hash and byte count. Because the importance-matrix path is embedded in the GGUF metadata, the byte count of a re-quantization depends on where you put the imatrix file — the weights do not. Exact-size behaviour is scoped to the recorded source metadata and the recorded llama.cpp build; changing the converter, runtime, source layout or metadata requires revalidation.
Apache-2.0, inherited from the base model — follow the upstream license and model-card requirements.
XHToken — Spark2.6-4B · llama.cpp — quantization and the KL/perplexity evaluator · abliterix — the local input-gated ablation pipeline · FIT-GGUF — verified size-exact quantization. FIT-GGUF is an independent project, not affiliated with XHToken or ggml-org.
⭐ FIT-GGUF on GitHub — the tool, the method, the full research recordEnglish · 简体中文