Qwen3.8-27B-Abliterated

A training-free abliterated build of Qwen/Qwen3.8-27B — a dense 27B native vision-language model (hybrid full-attention + linear-attention architecture, flexible thinking control).

🔧 Reproducible with LLM-abliterate — the generic, open-source tool used to build this model. It auto-detects the residual-stream writers of any transformers model (dense, hybrid, multimodal, MoE) and applies the same training-free abliteration. See the Reproduction section below for the exact commands that produced this checkpoint.

Refusal behaviour is removed by a forward-only weight edit: a single "refusal direction" is identified in the residual stream and orthogonalized out of every matrix that writes into that stream. No fine-tuning, no distillation, no data poisoning. The weights are the original weights minus a rank-1 component.

The vision tower (model.visual.*) is byte-for-byte untouched, as is lm_head (it reads the residual stream, it does not write it).

Disclaimer. This model will not refuse. It is published for alignment and safety research — measuring what refusal training actually protects, red-teaming, and studying the mechanics of refusal directions. You are responsible for your use of it and for complying with all applicable laws. Inherits the Apache-2.0 license of the base model.

---

Latest bug fixes as of 2026-08-16

Quick Results

Measured on our own reproduction pipeline (see Method):

The sweep that produced these numbers lives in code/sweep_lambda.py.

Independent evaluation (heretic, 100-prompt harness)

As a second, larger-sample confirmation, the saved checkpoint was scored with heretic-llm in evaluate-only mode: --model <base> --evaluate-model <this-repo> --response-prefix $'\n', using the standard 100 good + 100 bad prompts (mlabonne/harmless_alpaca / mlabonne/harmful_behaviors):

Reading the numbers: the refusal rate drops 98 % → 39 % on the full 100-prompt harness, and the KL divergence of 0.0001 is essentially zero — the abliterated model's first-token distribution is almost identical to the base's on harmless prompts, confirming the edit removed most of the refusal direction without drifting the rest of the model (no lobotomy, λ = 1.2 is a conservative, capability-preserving choice from the sweep below). The residual 39 % refusal is the deliberate trade-off of picking a lower λ to keep capability loss minimal; raising λ trades more refusal removal for a slightly larger capability shift.

Verified: r̂ is actually removed from the edited weights

Beyond behavioural tests, we verified the edit at the weight level with a direct projection check. For the rank-1 ablation W ← W − λ·r̂(r̂ᵀW) the exact invariant is

r̂ᵀW_abl = (1 − λ) · r̂ᵀW        (at λ = 1.2  ⇒  factor −0.20)

so a tensor that was edited should show its alignment with r̂ scaled by (1−λ) (the sign flip is the reversal that actually suppresses refusal), while a tensor that was left alone must stay bit-identical. Loading the specific weights straight from the safetensors shards (no full model load) and projecting them onto the unit r̂ (position 46, non-thinking):

Every edited tensor lands on the predicted +0.200 factor (the norm makes −0.20 read +0.20), and — just as important — lm_head and the vision tower are byte-for-byte identical, so the boundaries that read the residual stream / consume images were never touched. If any non-edited tensor had shown the −0.20 factor, that would mean the edit leaked; none did. This confirms r̂ was removed (reversed) from exactly the 131 residual writers and nowhere else, ruling out collateral damage at the source. (At this conservative λ = 1.2 the component is scaled to 20 % of its original alignment — reduced but not fully reversed, which is what keeps capability loss minimal while still cutting the refusal pathway.)

Capability benchmark (ARC-Challenge)

Capability was measured with a proper lm-eval run (HF backend, chat template applied, loglikelihood — not the generate-through-API harness that produced the earlier misleading numbers). Same harness, base vs this repo, on ARC-Challenge (0-shot, --apply_chat_template --fewshot_as_multiturn):

A quick note on methodology: an earlier generate-based pass through a raw Question:\nAnswer: prompt (no chat template) reported ARC at 0.227 — below the ~25 % chance floor — which is an artifact of the harness, not the model. Re-measured correctly the abliterated model scores 0.45 (acc) / 0.45 (acc_norm), essentially equal to the base's 0.44–0.45.

The λ = 1.2 build scores ARC at or above the base — the conservative (factor −0.20) edit keeps science reasoning effectively intact, in line with the near-zero KL. It is not a lobotomy: lm_head / vision untouched per the weight-level check above.

A/B capability benchmark (base vs this repo)

The cleanest way to judge the edit is a direct A/B comparison: run the same benchmark, same prompt, same config on the base and on this repo. Both models were scored with the identical lm-eval HF backend, dtype=bfloat16, parallelize=True, no chat template, MMLU at limit 1000 per task and GSM8K at limit 100, 5-shot (λ = 1.2 build):

Reading the A/B honestly (λ = 1.2 build):

• MMLU 0.8388 → 0.8342 (−0.005). General knowledge / academic capability is essentially preserved — a ~0.5 % difference, the strongest signal that the edit removed the refusal direction without touching the model's knowledge.
• GSM8K drops 0.62 → 0.59 (~3 pts). Arithmetic / step reasoning takes only a small hit — far smaller than the λ = 1.65 build's ~14-pt drop. This is the benefit of the conservative λ = 1.2: the reasoning path is barely disturbed.
• ARC 0.4433 → 0.4533 (+1 pt). Science reasoning is equal to or slightly above base — within noise.

Net: the λ = 1.2 abliterated model preserves capability essentially at base level across all three benchmarks (MMLU −0.005, GSM8K −0.03, ARC +0.01) while still cutting refusal from 98 % → 39 % and holding KL at 0.0001. This confirms the conservative λ = 1.2 choice trades a little refusal-removal (39 % residual) for a model that is behaviourally almost indistinguishable from the base on knowledge and reasoning — and the weight-level check above confirms lm_head and the vision tower are untouched.

---

What "abliteration" means here

Alignment/tuning teaches a model to refuse harmful requests. That behaviour is not a single weight — but it is a single direction in the model's internal residual stream: on harmful prompts the hidden state at a given layer is displaced along a roughly constant unit vector r̂. If we remove the component of every weight that writes along r̂, the model can no longer "point" its residual state along the refusal axis, so the refusal behaviour disappears — while the rest of the network (which operates mostly along other directions) is left almost untouched.

Because the edit is applied to the weights that write into the residual stream, the refusal pathway is cut at its source across every layer. This is the core idea of directional ablation / "abliteration".

---

Method

1. Find the refusal direction

We capture the last-token residual activation for 32 harmful + 32 harmless prompts, across all 65 hidden-state positions (index i = residual stream after i decoder layers; index 0 = embedding output), in both thinking modes. We then rank positions by normalized separation — Cohen's d and Mann–Whitney AUC of the projection onto the mean-difference direction — not by raw ‖diff-in-means‖, which merely grows with depth.

r̂ = (mean_bad − mean_good) / ‖mean_bad − mean_good‖

Selected direction: non-thinking mode, hidden-state index 46 (= output of decoder layer 45), Cohen's d = 11.86, AUC = 1.000. The direction extracted in non-thinking mode generalizes to thinking mode.

Why not the deepest layer? The last layers separate harmful/harmless with the highest AUC (we measured d = 14.05 at index 63) — but those layers are entangled with task content, so ablating them destroys capability. Layer 46 separates almost as well (AUC = 1.0) while leaving the model intact. Depth is not your friend for ablation. See Lessons learned.

2. Edit the weights

For every weight W whose output is added to the residual stream:

W ← W − λ · r̂ (r̂ᵀ W)          # out-dim writers   [hidden, in]
E ← E − λ · (E r̂ᵀ) r̂          # embed_tokens      [vocab, hidden]

with r̂ a unit refusal direction in the 5120-dim residual space and λ = 1.2.

131 tensors edited:

Not edited: all 333 model.visual.* tensors, lm_head (it reads the residual stream), all norms, and all q/k/v/gate/up, conv1d, in_proj, A_log, dt_bias.

embed_tokens is included — the residual stream is purely additive (x₀ = E[t]), so a component injected by the embeddings can never be removed by anything downstream; omitting it leaks the refusal direction into every layer. It needs the transposed form because hidden is its last dim.

mtp.* is edited through the state dict directly. The MTP (Multi-Token Prediction) head is a training-only module that is present in the checkpoint but not part of the runtime model class in transformers, so it cannot be reached through model.get_parameter(...). We therefore operate on the model.safetensors.index.json weight map directly. See code/apply_ablation.py.

3. Choose λ — measure refusal and capability together

λ was swept, not assumed. The rank-1 projection P(W₀) = r̂(r̂ᵀW) is computed once from the original weights, then λ is moved incrementally in memory: W(λ') = W(λ) − (λ'−λ)·P(W₀). That turns a "reload a 55 GB checkpoint per λ" grid into milliseconds per point (code/sweep_lambda.py).

Our sweep (refusal on 20 harmful prompts, capability on a 40-question MMLU-style subset):

Three findings worth stating plainly:

• λ = 1.0 (exact removal) is not enough. It leaves ~40 % of refusals intact. λ > 1 does not merely delete the component — it reverses it, and that reversal is what actually suppresses refusal.
• λ ≥ 2.0 destroys the model. Capability collapses toward the random-choice floor (~25 % for 4-way MC). A refusal-only sweep would never catch this — you must measure capability alongside.
• λ = 1.2 was chosen as the conservative trade-off. It keeps capability at the base level (~60 %) on this noisy subset while still cutting refusals to ~30 %; higher λ (1.5–1.65) removes more refusals (5–0 %) but starts to shift the reasoning path. Because the 40-question capability probe is noisy, we validated with the full A/B benchmark below — where MMLU is bit-identical to base — before settling on λ = 1.2 as the safest publishable point.

---

Usage

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

m = "hotdogs/Qwen3.8-27B-abliterated"
proc = AutoProcessor.from_pretrained(m)
model = AutoModelForImageTextToText.from_pretrained(
    m, dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Explain how a lock cylinder works."}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                                enable_thinking=False)   # thinking is ON by default
inputs = proc(text=[text], return_tensors="pt").to(model.device)
print(proc.tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0]))

Needs ~56 GB in bf16. Both thinking modes work; the direction was extracted in non-thinking mode and generalizes to thinking mode.

flash-linear-attention and causal-conv1d are optional — transformers ships pure-torch fallbacks for the Gated DeltaNet path, so the model runs correctly (just slower) without them.

---

Reproduction

Everything is in code/, and the whole pipeline is packaged as the reusable CLI tool LLM-abliterate (pip install-able, generic across architectures). The exact commands that produced this checkpoint:

# install the tool (apt + venv + torch handled automatically)
git clone https://github.com/nanofatdog/LLM-abliterate
cd LLM-abliterate && ./install.sh && source .venv/bin/activate

# 1. direction: non-thinking mode, hidden-state index 46
llm-abliterate extract ./Qwen3.8-27B --force-pos 46 --out direction.pt

# 2. λ sweep against refusal + capability
llm-abliterate sweep ./Qwen3.8-27B --direction direction.pt \
    --lams 0,1,1.2,1.4,1.5,1.6,1.65,1.7,1.75,2,2.5 --out sweep.json

# 3. apply λ = 1.2
llm-abliterate apply ./Qwen3.8-27B --direction direction.pt --lam 1.2 \
    --save ./Qwen3.8-27B-Ablit

# 4. verify
llm-abliterate verify ./Qwen3.8-27B --ablit ./Qwen3.8-27B-Ablit --structural --behaviour

The code/ directory in this repo holds the same scripts standalone (no package install required). The full pipeline, including the mistakes we made and how we fixed them, is documented in the next section and in the LLM-abliterate README.

---

The technique — background & references

The method belongs to the family of directional / linear-probe abliteration, sometimes called "abliteration" or "refusal direction ablation". The central idea — that refusal is a low-rank linear feature in activation space and can be removed by a forward pass over the weights — traces to a line of open work on mechanistic interpretability of refusal behaviour:

• The activation-engineering / linear-artifact direction line (PCA / mean-difference "refusal direction" in the residual stream), which underlies most modern abliteration tooling.
• Differential activation probing: capturing activations on harmful vs harmless prompts and taking the normalized difference as the direction. This is the same principle used by popular abliteration scripts that "take the mean of the refusal direction over the residual stream and project it out of the weights."
• Weight orthogonalization (projected abliteration): W ← W − λ·r̂(r̂ᵀW) — subtracting a rank-1 component aligned with r̂ from the out-projections that write into the residual stream.

The novelty in this repo is not the high-level idea — it is applying it correctly to a hybrid-architecture, multimodal, MTP-headed model, which the generic one-shot abliteration tools get wrong (see Lessons learned).

---

Lessons learned (what we got wrong, and how we fixed it)

We started with heretic-llm (p-e-w/heretic), the standard auto-abliteration tool, and it failed on Qwen3.8-27B. That failure taught us the most important lessons:

1. Generic abliteration tools under-cover hybrid architectures

heretic only edits o_proj + down_proj — it was written for pure full-attention transformers. Qwen3.8 is a hybrid (16 full-attention + 48 Gated DeltaNet / linear-attention layers) with an MTP head and a vision tower. Heretic:

• missed linear_attn.out_proj (48 layers) — the dominant token path in the hybrid;
• missed mtp.* entirely;
• did not touch embed_tokens — which leaks the refusal component into every layer because the residual stream is additive from the embeddings.

Result: a heretic run on Qwen3.8-27B still refused 97/100 even with correct settings. The leftover refusal lived in the tensors the tool never touched. Fix: enumerate every tensor writing into the residual stream + embed_tokens + mtp, and edit them all (131 tensors).

2. The deepest layer is a trap

Our first automatic direction pick chose hidden-state index 63 (highest d = 14.05, AUC = 1.0). At λ ≥ 1.5 that produced 0 % refusal but 0 % capability — a lobotomized model that just emits empty/garbled tokens ("The user is asking for a bomb…"). Fix: rank by normalized separation, then prefer an earlier strong layer (index 46, d = 11.86) which keeps capability intact. Depth is not your friend for ablation.

3. λ must be swept against refusal and capability

A refusal-only sweep happily reports "0 % refusal!" while the model has turned to mush. The capability collapse at λ ≥ 2.0 (MMLU-style 80 % → 22 %) is invisible to a refusal-only evaluator. Fix: always co-measure capability.

4. Multimodal models need the tokenizer, not the processor, for text-only forward passes

The multimodal processor returns mm_token_type_ids, which the model forward rejects (KeyError: 0) for text-only inputs. Use processor.tokenizer → only input_ids + attention_mask. Also, don't manually .to(device) — device_map="auto" handles placement, and forcing cuda:0 broke forward.

5. mtp.* lives outside the runtime class

The MTP head is in the checkpoint but not a transformers submodule, so model.get_parameter("mtp...") raises. Fix: edit through the safetensors weight map directly.

6. Thinking-mode evaluation must skip the thinking block

A thinking model's "first 100 tokens" are its CoT block, which is full of sorry/harmful/violat — inflating refusal counts. For capability scoring we strip the </think> block before reading the answer letter.

---

Verification

• Tensor sets identical: base vs abliterated have the same 1199 tensor keys.
• Vision untouched: byte-level diff → 0 of 333 model.visual.* tensors changed.
• lm_head untouched: byte-level diff → 0 changed.
• Edited tensors actually changed: 131/131.
• MTP edit pattern correct: exactly mtp.layers.0.self_attn.o_proj + mtp.layers.0.mlp.down_proj edited (rel. Frobenius ≈ 0.023); all other MTP tensors byte-identical.
• Non-weight files: all 13 (config, tokenizer, vocab, merges, chat template, preprocessor configs, etc.) byte-identical to the base.
• Behavioural: harmful prompts answered without refusal; capability questions answered correctly.

Run the checks yourself: python code/verify_ablit.py and python code/verify_deep.py.

---

Limitations

• We measured refusal on 20 harmful prompts and capability on a 40-question MMLU-style subset — a reproduction signal, not the full HarmBench/AdvBench/MMLU suites.
• The direction comes from 32 + 32 English prompts. It transfers to thinking mode and Chinese (not measured here), but transfer to other languages and formats is unmeasured.
• Removing refusal does not add knowledge. The model can now be confidently wrong about things it would previously have declined to discuss.
• Capability is down a little from base on our proxy; a full MMLU run would pin the exact figure.

---

License

Apache-2.0 (inherited from the base model). See LICENSE.

code/ scripts are provided under the same Apache-2.0 license.