Qwythos-9B-v2-Heretic-GGUF
GGUF quantizations of WaveCut/Qwythos-9B-v2-Heretic — the Heretic-decensored version of empero-ai/Qwythos-9B-v2. Use these with llama.cpp, Ollama, LM Studio, KoboldCpp, or any other GGUF runtime.
Files
| File | Format | ~Bits/weight | Size | Recommended use |
|---|---|---|---|---|
Qwythos-9B-v2-Heretic.Q4_K_M.gguf |
K-quant (mixed) | 4.5 | ~5.6 GB | Best 4-bit default — fits in ~6.5 GB VRAM, balanced quality/size |
Qwythos-9B-v2-Heretic.Q5_K_M.gguf |
K-quant (mixed) | 5.5 | ~6.5 GB | Higher quality 5-bit, ~7.5 GB VRAM |
Qwythos-9B-v2-Heretic.Q6_K.gguf |
K-quant (mixed) | 6.6 | ~7.4 GB | Very close to FP16, ~8.5 GB VRAM |
Qwythos-9B-v2-Heretic.Q8_0.gguf |
8-bit symmetric | 8.5 | ~9.5 GB | Effectively lossless, ~10.5 GB VRAM |
Quantization recipe
| Step | Tool | Version |
|---|---|---|
| HF → F16 GGUF | convert_hf_to_gguf.py |
llama.cpp b9986 (commit 91c631b), run with --no-mtp |
| Quantize | llama-quantize |
llama.cpp b9986 (prebuilt linux-x64) |
Note on --no-mtp: the base Qwen3.5 model bundles a multi-token-prediction (MTP) head as block #32. Excluding it via --no-mtp produces a clean 32-block text GGUF that loads correctly in current llama.cpp builds. If you want speculative decoding with the MTP head, run convert_hf_to_gguf.py --mtp separately.
No imatrix (importance matrix) calibration was used — K-quants hold up well without one for this model size, and Q8_0 is format-defined and never benefits from it.
Architecture
Qwen3.5 hybrid — 32 blocks mixing attention and SSM (Mamba-style) layers. Supported in llama.cpp b9986 and later under the qwen35 architecture key.
Usage
# llama.cpp
llama-cli -m Qwythos-9B-v2-Heretic.Q4_K_M.gguf -p "Hello" --chat-template chat_template.jinja
# Ollama (create a Modelfile pointing to the .gguf)
ollama create qwythos-heretic -f Modelfile
ollama run qwythos-heretic
Disclaimer
Uncensored (safety alignment removed via Heretic). The original empero-ai/Qwythos-9B-v2 maintainers are not affiliated with this derivative. Use responsibly.