Qwen3.8-27B-abliterated GGUF
imatrix GGUF quants of rajaykumar12959/qwen3.8-27b-abliterated, a refusal-ablated Qwen3.8-27B, sized to run on a 16 GB GPU with llama.cpp. Each quant was evaluated against the BF16 original. The scripts, raw outputs and full report are at github.com/vedevpatel/qwen3.8-quantized.
This model has had its refusal behaviour removed. It will answer requests the original Qwen3.8 declines. You are responsible for how you use it. See the source model card for how the ablation was done and what it costs.
Files
| File | Size | MMLU-Pro (Δ vs BF16) | Mean KLD vs BF16 | Pick it if |
|---|---|---|---|---|
Qwen3.8-27B-abliterated-IQ3_M.gguf |
12.6 GB | 79.00% (−0.60 pp) | 0.0642 | You have 16 GB of VRAM: up to 32K context, or 64K with an 8-bit KV cache |
Qwen3.8-27B-abliterated-IQ4_XS.gguf |
15.1 GB | 79.64% (+0.05 pp) | 0.0195 | You want higher fidelity: on 16 GB, up to 8K context (16K with an 8-bit KV cache) |
Qwen3.8-27B-abliterated-Q4_K_M.gguf |
16.5 GB | 78.83% (−0.76 pp) | 0.0167 | You have more than 16 GB of VRAM (not tested on a 16 GB card) |
imatrix.gguf |
27 MB | You want to make your own quants |
BF16 scores 79.60% on the same questions. No quant differs significantly from BF16 on MMLU-Pro (all 95% intervals include zero). KL divergence on WikiText-2 is the more sensitive measure: IQ3_M drifts 3.3–3.8× more than the 4-bit quants, so prefer a 4-bit file for open-ended generation if it fits.
MMLU-Pro here is 4,200 questions (300 per category), 5-shot chain of thought, greedy, thinking off. These absolute scores aren't comparable to the 61.8% on the source model card, which used a different setup; only the comparisons between files are like-for-like. Full report.
Speed
On an RTX 4070 Ti Super 16 GB, with the whole model on the GPU (llama-bench -ngl 99, pp512/tg128):
| Quant | Generation (tok/s) | Prompt processing (tok/s) | Max context at full speed | With 8-bit KV cache |
|---|---|---|---|---|
| IQ3_M | 39.9 | 1,772 | 32K (34.6 tok/s) | 64K (29.9 tok/s) |
| IQ4_XS | 35.1 | 1,571 | 8K (34.3 tok/s) | 16K (32.9 tok/s) |
Beyond these context sizes the KV cache spills out of VRAM and speed drops sharply. The limits were measured with 0.6 GB of VRAM in use by the desktop, so they shrink if other programs hold more. The 8-bit cache (-fa on -ctk q8_0 -ctv q8_0) costs almost no speed; its effect on output quality wasn't measured. Q4_K_M wasn't tested on this card.
On an A100 80 GB, generation runs at 47.6 tok/s (IQ3_M), 59.5 (IQ4_XS) and 48.1 (Q4_K_M).
Running it
llama-server -m Qwen3.8-27B-abliterated-IQ3_M.gguf -ngl 99 -c 32768 --jinja --port 8080
Then open http://127.0.0.1:8080, or use the OpenAI-compatible API at /v1. For up to 64K of context on 16 GB, add -fa on -ctk q8_0 -ctv q8_0 and set -c 65536. With IQ4_XS on 16 GB, use -c 8192 (or -c 16384 with the 8-bit cache).
The chat template puts the model in thinking mode by default, so it reasons at length before every answer. For direct answers, start the server with --reasoning off, or send "chat_template_kwargs": {"enable_thinking": false} with a request. llama-cli/llama-completion don't accept template kwargs, so thinking stays on there.
Text input only: the source checkpoint has no vision encoder.
How these were made
- Source:
rajaykumar12959/qwen3.8-27b-abliterated@9f85b10, converted with llama.cpp @23f7aab. - Converted with
--no-mtp. The source config declares an MTP layer, but the checkpoint ships no MTP weights. tokenizer_config.jsonwas replaced with the one fromQwen/Qwen3.8-27B@1d4bf0f, because the source's copy can't be loaded by the converter. The tokenizer itself is identical to upstream.- The importance matrix was computed on a Q8_0 copy using bartowski's
calibration_datav3.
License
Apache-2.0, following the source model and Qwen3.8-27B.