Huihui-Qwable-3.6-27b-abliterated Q4_K_M (MTP, GGUF)
A Q4_K_M GGUF quant of huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF.
Why this quant exists: huihui-ai ships Q4_K_M_Q8 (19.1 GiB) and Q4_K_M_F16 (25.9 GiB). Both are a little too large to leave working room on a 22 GB card once you add the KV cache and compute buffers. This plain Q4_K_M is 15.7 GiB, so it keeps headroom on 22–24 GB GPUs (e.g. RTX 3090 / 4090, or a 22 GB-modded 2080 Ti) while keeping the baked MTP layer intact.
| File | Size | BPW |
|---|---|---|
Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf |
15.7 GiB | 4.92 |
Lineage
- Base: Mia-AiLab/Qwable-3.6-27b — Qwen3.6-27B with Claude Fable-5 reasoning and assistant traces distilled via full SFT.
- Abliteration + MTP: huihui-ai.
- This quant:
Q4_K_M, produced with llama.cpp (buildb9190) from huihui'sQ4_K_M_F16release.
Speculative decoding (MTP)
The MTP draft head is baked into the model (the nextn layer), so there is no separate draft file. Run it as self-speculation:
llama-server -m Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf \
-ngl 99 -fa on -c 32768 --parallel 1 --jinja \
--spec-type draft-mtp --spec-draft-n-max 3
Tip: --spec-draft-n-max 3 works better than the larger value suggested upstream. At higher settings the draft head proposes too many tokens, acceptance drops, and throughput ends up lower. Measured on a GB10 (Grace-Blackwell):
--spec-draft-n-max |
decode | draft acceptance |
|---|---|---|
3 |
~18.7 tok/s | ~0.45 |
6 |
slower | ~0.25 |
Acceptance is lower than that of a vanilla Qwen3.6 (~0.75). That gap appears tied to the Fable distill rather than the quantization: changing the draft-head precision (Q4, Q8, or F16) did not materially affect acceptance in testing.
Notes
- Tool calling works with the standard OpenAI-compatible
toolsschema (it returns structuredtool_calls) when thinking is disabled — e.g. pass"chat_template_kwargs": {"enable_thinking": false}in the request. With thinking enabled, the Fable traces can surface internal tool names in the reasoning output. - This is an abliterated model.
- License inherited from the base: MIT.
Credits
Thanks to Mia-AiLab for the Fable distill and huihui-ai for the abliterated MTP build.
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release 557af03fbfc35ccc237abeb0d5ee7eaa767b62be, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF | not recorded | Immediate Q4_K_M_F16 GGUF input |
| Mia-AiLab/Qwable-3.6-27b | not recorded | Upstream SFT/distilled model lineage |
Build details
Tools and versions: llama.cpp build b9190, as recorded in the existing card; exact git commit and quantizer invocation not recovered.
The Q4_K_M output was produced from Huihui’s Q4_K_M_F16 release, retaining the baked-in MTP nextn layer. The source was already quantized with mixed precision; this was not a fresh quantization of original BF16 weights.
No additional SFT or weight merging by coolthor is documented for this conversion. The upstream Qwable model already contains SFT/distillation and Huihui abliteration/MTP modifications; this provenance does not erase that training history.
Exact build command: not recorded in the recovered material. The method description above is not a complete executable reproduction recipe.
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf |
16810713696 | 1e95a76a4816afca0b3c493202200e1ccd27a66983ca1b019270c8e17249619d |
Verifying the files
Download the desired files from revision 557af03fbfc35ccc237abeb0d5ee7eaa767b62be and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Immediate source revision and exact source filename; Exact tool git commit and invocation; Per-tensor conversion/copy map.
Sources: existing model card and HF metadata.