Sign up

coolthor/Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP-GGUF

🤗 On Hugging Facemit17 GBGGUF✓ Checksums witnessedupdated today
Magnet

Huihui-Qwable-3.6-27b-abliterated Q4_K_M (MTP, GGUF)

A Q4_K_M GGUF quant of huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF.

Why this quant exists: huihui-ai ships Q4_K_M_Q8 (19.1 GiB) and Q4_K_M_F16 (25.9 GiB). Both are a little too large to leave working room on a 22 GB card once you add the KV cache and compute buffers. This plain Q4_K_M is 15.7 GiB, so it keeps headroom on 22–24 GB GPUs (e.g. RTX 3090 / 4090, or a 22 GB-modded 2080 Ti) while keeping the baked MTP layer intact.

| File | Size | BPW |

|------|------|-----|

| Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf | 15.7 GiB | 4.92 |

Lineage

  • Base: Mia-AiLab/Qwable-3.6-27b — Qwen3.6-27B with Claude Fable-5 reasoning and assistant traces distilled via full SFT.
  • Abliteration + MTP: huihui-ai.
  • This quant: Q4_K_M, produced with llama.cpp (build b9190) from huihui's Q4_K_M_F16 release.

Speculative decoding (MTP)

The MTP draft head is baked into the model (the nextn layer), so there is no separate draft file. Run it as self-speculation:

llama-server -m Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf \
  -ngl 99 -fa on -c 32768 --parallel 1 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 3

Tip: --spec-draft-n-max 3 works better than the larger value suggested upstream. At higher settings the draft head proposes too many tokens, acceptance drops, and throughput ends up lower. Measured on a GB10 (Grace-Blackwell):

| --spec-draft-n-max | decode | draft acceptance |

|---|---|---|

| 3 | ~18.7 tok/s | ~0.45 |

| 6 | slower | ~0.25 |

Acceptance is lower than that of a vanilla Qwen3.6 (~0.75). That gap appears tied to the Fable distill rather than the quantization: changing the draft-head precision (Q4, Q8, or F16) did not materially affect acceptance in testing.

Notes

  • Tool calling works with the standard OpenAI-compatible tools schema (it returns structured tool_calls) when thinking is disabled — e.g. pass "chat_template_kwargs": {"enable_thinking": false} in the request. With thinking enabled, the Fable traces can surface internal tool names in the reasoning output.
  • This is an abliterated model.
  • License inherited from the base: MIT.

Credits

Thanks to Mia-AiLab for the Fable distill and huihui-ai for the abliterated MTP build.

📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.