FIT-GGUF v0.5.1ALL FIVE TIERS GATE-VERIFIED5 FIT + BF16 + IMATRIXMTP · VISION + VIDEOMEASURED KL + SAME-TOPAPACHE-2.0

Sage-A1.5-abliterated · FIT-GGUF

Five fidelity tiers of a 35B MoE agentic model — each one the smallest GGUF that provably meets its KL gate, re-measured on its own shipped bytes.

11.11 GiB · MINIthe verified minimum at each fidelity66.19 GiB · BF16

English · 简体中文 📖

🧭 About FIT-GGUF — the tool behind these files

Every FIT tier here was planned, executed and verified by FIT-GGUF, an open-source, deterministic tensor-level planning layer on top of llama.cpp quantization. Standard GGUF quantization asks you to pick one of a handful of presets; FIT-GGUF instead asks what quality do you want, then finds and verifies the smallest GGUF that demonstrably meets it.

Traditional GGUF gives you presets. FIT gives you a fidelity contract: macro KL ≤ tier anchor, measured against this model's own BF16 on five fixed domains.

Method, pre-registered research record and the fit CLI are all open source: github.com/Scorp1o117/FIT-GGUF

⚠️ Safety notice / 安全提示

The source model is abliterated, refusal-removed and has no meaningful built-in guardrails. It may comply with harmful, illegal or unsafe requests. Use it only where you can provide appropriate moderation, access control and legal review. Do not deploy it to end users without your own safety layer.

源模型经过拒答方向消融，不具备可靠的内置安全护栏；请仅在合法、受控、具备审核与滥用防护的环境中使用，使用者自行承担部署责任。

🧬 Abliteration — measured effect

The refusal direction was ablated locally with the abliterix pipeline: expert-side suppression n_suppress=12, expert_ablation_weight=8.25, across all 40 layers. The table is a strict single-variable comparison — abliterated BF16 against the original base BF16, quantization plays no part.

Two KLs — do not mix them. The 0.011436 above is the abliteration's perturbation of the base model. Every other KL on this page is quantization divergence against this repository's BF16. They are measured on different objects and are not comparable.

📦 Everything in this repository

Five FIT tiers and the BF16 reference — every file measured by the same evaluator, on the same five slices, against the same reference logits. Pick by VRAM: balanced at 13.17 GiB is the usual sweet spot, mini at 11.11 GiB fits a 12 GB card, reference at 23.61 GiB fits 24 GB.

All five tiers pass 5/5 domains. Every shipped file was re-evaluated on its own bytes after quantization and reproduced its stated KL (maximum deviation 0.0009).

⚡ MTP and vision

MTP works — +35% generation, measured

Every file keeps the MTP layer (nextn_predict_layers=1) and it is enabled directly. Same model, same prompt:

llama-cli -m <file>.gguf -ngl 38 -p "Hello" -n 64 # 28.4 t/s llama-cli -m <file>.gguf -ngl 38 -p "Hello" -n 64 --spec-type draft-mtp # 38.4 t/s

Your runtime needs --spec-type draft-mtp support (b11612 or newer).

Vision + video come from mmproj

The source model is multimodal (a 27-layer vision tower). The main GGUF contains no vision tensors, so vision and video only work through mmproj:

llama-cli -m <file>.gguf --mmproj mmproj-Sage-A1.5-abliterated-bf16.gguf -ngl 38 # log: modalities : text, vision, video

Without mmproj the loader reports modalities : text and you get text only.

📈 Measured quality

llama.cpp b11612 · c=512 b=512 · five domains · against the BF16 in this repo. Native ladder points (GiB / macro KL):

Read the curve, not just the tiers. The gates are capped at KL 0.02–0.20, so the big native presets are cleaner than every tier — if you want maximum fidelity rather than minimum size, take Q6_K or the BF16 file.

🚀 Run it

Text only

./llama-server -m Sage-A1.5-abliterated-MTP-FIT-balanced-13.17GiB-IQ3_S.gguf -ngl 38 -c 32768

Vision + video

./llama-server -m Sage-A1.5-abliterated-MTP-FIT-balanced-13.17GiB-IQ3_S.gguf \ --mmproj mmproj-Sage-A1.5-abliterated-bf16.gguf -ngl 38 -c 16384

MTP acceleration

./llama-server -m <file>.gguf --spec-type draft-mtp -ngl 38 -c 16384

Recommended sampling (same as the source model): temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0

Context length: natively 256K (context_length = 262144). The upstream card's 512K requires YaRN (factor 2.0) and was not measured for this release — start at -c 32768 and work up. 30 of the 40 layers are linear attention (SSM), so the KV cache grows far more slowly than a pure full-attention model.

-ngl: on a 32 GB card the quantized tiers take -ngl 99 comfortably. The 66 GiB BF16 file does not — it needs --fit on --load-mode none instead of a hard -ngl.

🔬 Evaluation protocol & honest scope

Tier verification: a FIT tier ships only if macro KL ≤ its anchor. Each shipped file was re-evaluated on its own bytes after quantization and reproduced its stated KL.

The honest note on the reference tier: it is the native Q5_K_M preset rather than a FIT recipe — inside the [Q4_K_M, Q5_K_M] window no smaller reachable artifact clears 0.02, so the preset itself is the minimum passing point there. The other four tiers are genuine tensor-level FIT recipes.

🧩 Included — and not included

✅ 5 gate-verified FIT tiers · ✅ abliterated BF16 reference GGUF (the weights every measurement above is taken against) · ✅ the importance matrix used for every quantization (500×512 chunks) · ✅ mmproj (vision + video, BF16) · ✅ MTP layer in every artifact · ✅ per-domain raw data (results/measurements.json) · ✅ three FIT recipes (recipes/) · ✅ checksums (SHA256SUMS.txt)

❌ no Q6_K / Q8_0 tiers — for maximum fidelity take the BF16 file, or quantize it yourself with the bundled imatrix

🔍 Verify & reproduce

sha256sum -c SHA256SUMS.txt

Both inputs needed for reproduction ship here: Sage-A1.5-abliterated-MTP-BF16.gguf (source) and Sage-A1.5-abliterated-MTP-BF16-imatrix.gguf (the 500-chunk matrix). The *-tensor-types.txt files under recipes/ reproduce the corresponding tier:

llama-quantize \ --imatrix Sage-A1.5-abliterated-MTP-BF16-imatrix.gguf \ --tensor-type-file recipes/fit-balanced-tensor-types.txt \ Sage-A1.5-abliterated-MTP-BF16.gguf out.gguf IQ3_XXS 16

The base preset must match the recipe — see recipes/*-plan.json for each tier's lower_preset.

⚠️ Known limitations

· Two KLs, not interchangeable: the ablation figure (0.011436) and the per-tier quantization KLs measure different objects.

· 512K context unmeasured: only the native 256K was exercised.

· Vision quality unmeasured: mmproj was verified to load; no vision benchmark was run.

· No Q6_K / Q8_0: use BF16, or quantize from the bundled inputs.

📄 License & credits

Apache-2.0, inherited from the base model — follow the upstream license and model-card requirements.

InternScience — Sage-A1.5 · llama.cpp — quantization and the KL/perplexity evaluator · abliterix — the local ablation pipeline · FIT-GGUF — verified size-exact quantization. FIT-GGUF is an independent project, not affiliated with InternScience or ggml-org.

⭐ FIT-GGUF on GitHub — the tool, the method, the full research record

English · 简体中文