Qwen3.8-27B-EXL3-3.5bpw-heretic

This is an abliterated build of Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw, made with Heretic running directly on the quantized EXL3 model.

The EXL3 3.5bpw weights are bit-identical to the original. Nothing was requantized or merged. The abliteration is stored as a small sidecar file (abliteration.safetensors, 2.9 MB). At inference time it is applied exactly to the outputs of the unmodified quantized projections.

How it works

Heretic abliterates each output projection W that writes into the residual stream: attention o_proj, Gated DeltaNet out_proj, and MLP down_proj. With norm-preserving row normalization (row_normalization = "full") the modified projection is

W' = diag(n / m) · (I − λ v vᵀ) · Wn

• Wn is W with each row normalized to unit length.
• n holds W's original row norms.
• m holds the row norms of (I − λ v vᵀ) Wn.

The whole modification depends only on the projection output y = W x:

W' x  =  y ⊙ s  −  b · ⟨y, a⟩        a = v / n,   b = λ · n ⊙ v / m,   s = 1 / m

a, b and s are precomputed per projection from the dequantized EXL3 weights and stored in abliteration.safetensors, under the keys <block>.abl_attn.{a,b,s} and <block>.abl_mlp.{a,b,s}. Applying them to the output of the original quantized matmul reproduces the abliterated projection exactly. Heretic's own export approximates the same change with a low-rank SVD before merging it into bf16 weights; this version does not.

Consequences:

• The quantized tensors are never touched, so the 3.5bpw quality/size trade-off stays exactly as shipped.
• Turn the sidecar off and you have the original model, bit-for-bit. This was verified through TabbyAPI: greedy generations were byte-identical.
• The runtime cost is 100 small fused ops per forward pass, about 1–2% of decode speed.

Usage

Stock exllamav3 (any backend, no rebuild)

from exllamav3 import Config, Model, Cache, Tokenizer, Generator
import exl3_ablation                      # ships in this repo

model_dir = "Qwen3.8-27B-EXL3-3.5bpw-heretic"
config = Config.from_directory(model_dir)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens = 32768)
model.load()
exl3_ablation.apply(model, model_dir)     # abliterated
# exl3_ablation.remove(model)             # back to the original model

exl3_ablation.py wraps the attention and MLP modules of every transformer block. It works on the fused and graph decode paths, and with speculative decoding (DFlash2 / MTP drafts).

TabbyAPI

TabbyAPI loads exllamav3 models itself. Add one line after the model is loaded in its exllamav3 backend: exl3_ablation.apply(self.model, model_dir). Alternatively, use an exllamav3 build that loads abliteration.safetensors natively. Without either, TabbyAPI serves the original, non-abliterated model: the weights are the same, and the sidecar is simply ignored.

Method details

• Tool: Heretic 2.0.0.dev0 (commit 3521f86). A custom model backend ran every Heretic forward pass (residuals, first-token logits, greedy generations) on the EXL3 model itself through exllamav3. Directions, KL divergence and refusals were therefore measured on exactly the model that gets deployed.
• Target: thinking mode (the chat template's default <think> reasoning). The model decides to refuse inside its reasoning, and ablation tuned only for non-thinking mode left 65% hard refusals when thinking was on.
• Refusal directions: standard Heretic. Per-layer mean residual difference between 400 harmful prompts (mlabonne/harmful_behaviors) and 400 harmless prompts (mlabonne/harmless_alpaca), taken at the position after a closed think block and orthogonalized against the harmless mean. Stored in abliteration_directions.safetensors with shape (65, 5120).
• Objectives: refusals were scored on the first 160 reasoning tokens with reasoning-style markers ("refuse", "decline", "offer an alternative", …). KL divergence was measured on the first reasoning token over 100 harmless prompts.
• Search: 200 Optuna TPE trials (60 random startup trials, seed 1337), plus 5 enqueued configurations from a manual sweep.
• Selected trial (198): refusals 4/100 (baseline 96/100), KL 0.021. The parameters are in abliteration.json. 100 projections are ablated.

Evaluation

The same weights were loaded in one process, with the ablation toggled on and off. All decoding was greedy. Raw numbers are in eval/.

Quality

Refusals, held out from the optimization

* Heretic's default keyword list also matches words like "illegal" and "harmful" inside answers that comply. Most of the remaining matches are of that kind, or are answers that add a disclaimer.

Reasoning health (thinking mode, up to 6144 reasoning tokens)

On harmful requests the model now actually works through the task instead of refusing after a short reasoning pass, so its reasoning gets longer. Some of those prompts need a generous thinking budget. Harmless prompts behave like the original.

Speed (TabbyAPI, RX 7900 XTX with ROCm, DFlash2 draft, Q8 KV cache, 512 greedy tokens)

Files

Disclaimer

This model will comply with requests that the original model refuses. You are responsible for how you use it and for following applicable laws. Base model: Qwen3.8-27B by the Qwen team, EXL3 quantization by Mia-AiLab. Both are under Apache-2.0.