Qwen3.8-27B, abliterated, with a 131,072-token window on one 16 GB GPU. The whole model and its cache stay on the card. Three things make that work:
UD-Q3_K_XL (Dynamic 3.1) weights. The 60 matrices the abliteration changed are swapped in, and every other tensor is Unsloth's, byte for byte.Every number on this page is measured on an RTX 5080 16 GB, and the records are in recipe/.
llama-server -m Qwen3.8-27B-ASCII-Condensed-Heretic-Q3_K_XL.gguf \
-c 112640 -ngl 99 -fa on -ctk q4_0 -ctv q4_0 -ub 128 -b 512 -np 1 \
-ot 'token_embd\.weight=CPU' --spec-type ngram-mod
-c 112640 leaves headroom beside a desktop. On a card with less else on it, the full 131072 fits. Each 16K of window costs 288 MiB, and leaving out -c lets --fit choose. Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} to turn it off for a request.
cudaMalloc failed: out of memorywith memory free? Stock llama.cpp allocates the weights as one 11 GiB block, which a card that also drives a desktop can refuse. Build with the two patches inrecipe/tools/.
| Profile | Window | Code tok/s, deep | Prose tok/s | Extra flags |
|---|---|---|---|---|
| long (default) | 131,072 | 111 at 115K | 59 | --spec-type ngram-mod |
| fast | 90,112 | 150 at 79K | 113 | --spec-type draft-mtp,ngram-mod -ctkd q4_0 -ctvd q4_0 |
| quality | 73,728 | 137 at 65K | 59 | -ctk q8_0 -ctv q8_0 (with -c 73728) |
blk.64.*) loads only when draft-mtp is asked for.| File | Size | sha256 |
|---|---|---|
Qwen3.8-27B-ASCII-Condensed-Heretic-Q3_K_XL.gguf: the model |
11.60 GiB | c0d48dc2…08ea79 |
Qwen3.8-27B-Heretic-ablation-LoRA-F16.gguf: the abliteration as a LoRA for Unsloth's file |
263 MiB | 5c2e6681…7bc3b |
Qwen3.8-27B-ASCII-Condensed-Heretic-IQ4_NL-MTP.gguf: the 24 GB variant with MTP (below) |
14.55 GiB | 751b7504…6ca086 |
kept-token-ids.json: the old id at each new position |
||
recipe/: build and measurement records, tools, patches |
Full hashes are in SHA256SUMS.
| Model | Reference | Mean KLD | 99th pct | Same top token | PPL |
|---|---|---|---|---|---|
| Unsloth 3.1 (stock) | base Q8_0 | 0.0835 ± 0.0014 | 1.521 | 91.75% | 3.801 |
| Unsloth 3.0 (stock) | base Q8_0 | 0.0854 ± 0.0014 | 1.562 | 91.80% | 3.793 |
| This file | abliterated Q8_0 | 0.0898 ± 0.0015 | 1.594 | 91.62% | 3.848 |
| Adapter on Unsloth 3.1 | abliterated Q8_0 | 0.0873 ± 0.0015 | 1.540 | 91.62% | 3.846 |
| The abliteration itself | base Q8_0 | 0.0169 ± 0.0003 | 0.289 | 96.21% | 3.651 |
This file is about 7% further from its reference than Unsloth's is from its own. The 60 grafted tensors use stock llama.cpp rounding, because Unsloth's method is not public. The vocabulary prune leaves ASCII next-token distributions unchanged: the top 20 tokens are identical (37/37 checks) and their log-probability gaps match to within 8e-7.
Refusals: per 100 prompts under three scorers (Heretic keywords / refusal statements / classifier)| Model | mlabonne refusal set | JBB refusal set | JBB benign set |
|---|---|---|---|
| Base Q8_0 | 100 / 100 / 98 | — | — |
| Unsloth 3.1 (stock) | 100 / 100 / 99 | 96 / 94 / 96 | 38 / 30 / 38 |
| Abliterated Q8_0 | 55 / 0 / 46 | 46 / 1 / 46 | 7 / 1 / 11 |
| This file | 53 / 0 / 44 | 46 / 1 / 41 | 7 / 0 / 9 |
| Adapter on Unsloth 3.1 | 52 / 1 / 44 | 44 / 2 / 44 | 4 / 0 / 7 |
Refusal-Classifier.Error from the q4_0 cache (mean KLD against a reference cache): 0.0128 at 2K and 0.0052 at 30K (reference f16), then 0.0051, 0.0049, 0.0060 and 0.0065 at 16K, 32K, 49K and 64K (reference q8_0). That is under a tenth of the weights' own error.
Recall: 40 seven-digit codes are planted through held-out text and all asked back in one question, along with 5 decoys.
| Cache | Prompt (tokens) | Recalled | Decoys answered |
|---|---|---|---|
| q4_0 | ~63K | 80/80 | 0/10 |
| q8_0 | ~63K | 80/80 | 0/10 |
| q4_0 | ~115K | 80/80 | 1/10 |
This tests exact lookup only, not reasoning across the window. A 120,973-token prompt was read at 1,152 tok/s.
Speculative decoding: is the output the same?Drafts are verified by the model, so drafting changes speed, not what the model chooses. On a GPU, batched verification rounds differently from one-at-a-time decoding. The first divergence from the plain run came at token 48, where the plain run's top two candidates were 0.035 nats apart. Each run repeats exactly.
Build: graft, prune, sourcesffn_down, 22 ssm_out and 8 attn_output, all in layers 26–55.output.weight (834 → 430 MiB) and token_embd.weight (521 → 269 MiB); nothing is re-encoded.| Part | Source |
|---|---|
| Weights | unsloth/Qwen3.8-27B-GGUF UD-Q3_K_XL 3.1 @ 4af51b35 |
| Abliteration | 0bserverx/Qwen3.8-27B-Heretic-Abliterated-Uncensored (Heretic ARA) |
| References | ggml-org/Qwen3.8-27B-GGUF Q8_0; the abliteration's Q8_0 |
| Engine | llama.cpp b11279 (649dcb1036fd), CUDA 13, sm_120, plus the two patches |
KLD is measured with llama-perplexity --kl-divergence over WikiText-2 test plus Python stdlib source. VRAM figures come from NVML.
To keep Unsloth's file untouched and switch the abliteration per request (scale 1 abliterated, scale 0 the original):
llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf --lora-scaled Qwen3.8-27B-Heretic-ablation-LoRA-F16.gguf:1 \
-ngl 99 -fa on -ctk q4_0 -ctv q4_0 -ub 128 -b 512 -np 1 -ot 'token_embd\.weight=CPU'
7b20f067…5723.kept-token-ids.json maps them.Qwen3.8-27B-ASCII-Condensed-Heretic-IQ4_NL-MTP.gguf is the same abliterated model built for a 24 GB card instead of 16 GB. It's quantized from the abliterated full-precision weights rather than grafted onto Unsloth's file, and it carries Qwen3.8's multi-token-prediction head.
It's built for cards without tensor cores, such as the Tesla P40 (Pascal, sm_61). On those cards, i-quants like IQ3_S and IQ3_XXS decode through codebook lookups and spend more time on arithmetic than on memory reads. IQ4_NL keeps Q4_0's simple 32-weight block and adds a 16-entry table that fits in registers.
The build:
RVN-F16.gguf, the abliterated F16, to IQ4_NL with Unsloth's importance matrix. llama.cpp's usual rule raises attn_v and attn_qkv to Q5_K, 12% of the bytes.token_embd is Q8_0 and output is Q6_K.kept-token-ids.json).blk.64, nextn) is grafted from Unsloth's file: 351 MiB at Q6_K/Q8_0. It only drafts; every token it proposes is checked by the abliterated trunk.| File | Size | sha256 |
|---|---|---|
Qwen3.8-27B-ASCII-Condensed-Heretic-IQ4_NL-MTP.gguf |
14.55 GiB | 751b7504…6ca086 |
Fidelity: mean KLD against the abliterated Q8_0, the same 200 × 512 held-out tokens as the table above.
| Model | Mean KLD | 99th pct | Same top token | PPL ratio |
|---|---|---|---|---|
| IQ4_NL + MTP | 0.0447 ± 0.0008 | 0.786 | 93.80% | 1.024 |
| the 16 GB Q3_K_XL file above | 0.0898 ± 0.0015 | 1.594 | 91.62% | — |
Refusals: per 100 prompts under the same three scorers and protocol as above (Heretic keywords / refusal statements / classifier).
| Model | mlabonne refusal set | JBB refusal set | JBB benign set |
|---|---|---|---|
| IQ4_NL + MTP | 57 / 1 / 49 | 41 / 1 / 43 | 7 / 1 / 10 |
| Abliterated Q8_0 | 55 / 0 / 46 | 46 / 1 / 46 | 7 / 1 / 11 |
On the harmful sets, prompt by prompt, it doesn't differ from the abliterated Q8_0 under any scorer (exact McNemar p = 0.71 / 1.0 / 1.0).
Speed on one Tesla P40 24 GB: llama.cpp b11279, flash attention, q4_0 cache. Prompt processing:
| tok/s | |
|---|---|
| prefill, 512 / 2,048 tokens | 274 / 269 |
Decode, with --spec-type ngram-mod,draft-mtp against none:
| Workload | None | Thinking off | Thinking on |
|---|---|---|---|
| generation, short context | 15.2 | ||
| agent session (read, edit, debug, wrap up) | 15.6 | 47 | 24 |
| code rewrite (copies a file) | 15.0 | 115 | 22–24 |
| JSON / table | 15.4 | 40 | 33–37 |
| prose | 15.4 | 24 | 19 |
With 128K of q4_0 cache it needs about 18 GB: 14.6 GB of weights, about 2.4 GB of cache (16 full-attention layers) and about 1 GB of compute. It doesn't fit a 16 GB card at that window; use the Q3_K_XL file there.
llama-server -m Qwen3.8-27B-ASCII-Condensed-Heretic-IQ4_NL-MTP.gguf -ngl 99 -fa on \
-ctk q4_0 -ctv q4_0 -c 131072 --jinja --spec-type ngram-mod,draft-mtp
The abliteration removes learned refusals, so the model will answer requests the original declines. How you use it, and following the law where you are, is on you.
Apache 2.0, like every source. Credits: