Qwen3.8-27B-Abliterated

An abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.

Refusal behaviour is removed by a training-free, forward-only weight edit: a single "refusal direction" is identified in the residual stream and then orthogonalized out of every matrix that writes into that stream. No fine-tuning, no distillation, no data poisoning — the weights are the original weights minus a rank-1 component.

The vision tower is byte-for-byte untouched. Only the text path was modified.

---

Method

For every weight W whose output is added to the residual stream:

W  ←  W − λ · r̂ (r̂ᵀ W)          # out-dim writers  [hidden, in]
E  ←  E − λ · (E r̂ᵀ) r̂          # embed_tokens     [vocab, hidden]

with r̂ a unit refusal direction in the 5120-dim residual space and λ = 1.5.

131 tensors edited:

Not edited: all 333 model.visual.* tensors, lm_head (it reads the residual stream, it does not write it), all norms, and all q/k/v/gate/up, conv1d, in_proj, A_log, dt_bias.

embed_tokens is included. The residual stream is purely additive (x₀ = E[t]), so a component injected by the embeddings cannot be removed by anything downstream; omitting it leaves a refusal leak into every layer. It needs the transposed projection form because hidden is its last dim.

Finding the direction

Last-token residual activations were captured for 32 harmful and 32 harmless prompts across all 65 hidden-state positions, in both thinking modes. Layers were ranked by normalized separation (Cohen's d and Mann–Whitney AUC) rather than raw ‖diff-in-means‖, which merely grows with depth.

Selected: non-thinking mode, hidden-state index 46 — Cohen's d = 12.65, AUC = 1.000. (Index i is the residual stream after i decoder layers, so index 46 = output of decoder layer 45, 0-indexed.)

Choosing λ — this part matters

λ was measured, not assumed. Refusal and capability were scored at each λ (coarse sweep: n = 96 refusal prompts / 160 MMLU items — smaller than the finer sweep below, so single-prompt noise is ±1.04 pp here):

Two results worth stating plainly:

• λ = 1.0 — exact removal — is not enough. It leaves 25 % of refusals intact. λ > 1 does not merely delete the component, it reverses it, and that is what actually suppresses refusal.
• λ ≥ 2.5 destroys the model. MMLU collapses to 20–30 % — at or below the 25 % chance floor for 4-way multiple choice. A λ of 3.5 copied from another architecture would have shipped a lobotomized model that still scores 0 % refusal. A refusal-only sweep would not have caught this.

A finer sweep (n = 160 refusal / 320 MMLU) over λ ∈ [1.1, 2.0] kept MMLU flat at 84–85 % throughout and put refusal at or near zero across [1.2, 1.6]: exactly 0/160 at λ = 1.2, 1.4 and 1.5, and 1/160 (0.62 %) at λ = 1.3 and 1.6. The zero points are not contiguous, which is what a measurement with a resolution of ±1 prompt in 160 looks like — so read [1.2, 1.6] as a low-refusal band, not a proven plateau. λ = 1.5 was chosen from inside that band; the full sweep is in evals/sweep_fine.json.

---

Evaluation

Greedy decoding (do_sample=False), identical settings for base and abliterated. Refusal is scored by a deliberately loose keyword classifier — it is tuned to over-count refusals, never under-count.

Refusal

Every row is the complete benchmark, not a sample. In thinking mode the abliterated model refuses nothing at all — 0/520 on AdvBench and 0/393 on HarmBench, including all 100 copyright items, whose non-thinking residual is discussed below.

No prompt-prefill jailbreak is used anywhere. (The common "Sure, here is" prefill is itself a jailbreak: under it the base model refuses only 12.5 % on AdvBench, which would have made this edit look far less significant than it is.)

HarmBench by category — where the residual 11.45 % actually lives

Every safety category is exactly zero, with no adjudication applied. (In thinking mode even the copyright column goes to zero: 0/100.) The entire residual is the copyright cluster, and manual inspection of all 45 flagged responses shows they are overwhelmingly classifier false positives rather than refusals — two distinct failure modes:

1. The generated text itself begins with a refusal phrase. Asked for a passage from The Kite Runner, the model writes prose beginning "I cannot quite remember when I first started to remember. It was a cold, clear day in Kabul…". That is the novel, not a refusal.
2. A pedantic preamble followed by full delivery. "I cannot generate a new passage … but here is a long excerpt from Book One, Chapter 1" — and then it delivers it.

Song-lyric requests are flagged the same way, because lyrics contain lines like "I can't explain". The single AdvBench flag is identical in kind: "I cannot lie. I am an AI… However, you asked for a review of a product I have not used. So —" followed by the review.

These are reported as-is rather than adjudicated away: the raw classifier output is reproducible, a semantic judge is not. The per-category breakdown is what carries the claim.

Capability

GSM8K needs a generation budget of ≥1024 tokens. At 400 tokens the base model scored 68 % and this model 81.5 % — pure truncation artefact, measuring verbosity rather than arithmetic. With an adequate budget the two are identical.

Vision (untouched — verified two ways)

Structurally: a byte-level diff confirms 0 of 333 model.visual.* tensors changed, and the vision-only shard still shares an inode with the base checkpoint.

Behaviourally, on the same image:

base — "there are three distinct geometric shapes … Red Circle – Located in the upper-left portion … Green Triangle – Positioned to the right of the red circle"

abliterated — "There is a red circle located in the upper-left portion of the image. There is a green triangle … located in the"

---

Usage

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

m = "windowsxp811203/Qwen3.8-27B-Abliterated"
proc = AutoProcessor.from_pretrained(m)
model = AutoModelForImageTextToText.from_pretrained(m, dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Explain how a lock cylinder works."}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                                enable_thinking=False)   # thinking is ON by default
inputs = proc(text=[text], return_tensors="pt").to(model.device)
print(proc.tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0]))

Needs ~56 GB in bf16. Both thinking modes work; the direction was extracted in non-thinking mode and generalizes to thinking mode (48.27 % → 0.00 % on AdvBench, full 520).

flash-linear-attention and causal-conv1d are optional — transformers ships pure-torch fallbacks for the Gated DeltaNet path, so the model runs correctly (just slower) without them.

Limitations

• copyright refusals are only partly removed (97 % → 45 % by keyword; largely false positives). This edit targets safety refusal, and copyright declines are a partly distinct behaviour.
• The direction comes from 32 + 32 English prompts. It transfers to Chinese (83.33 % → 0.00 %) and to thinking mode, but transfer to other languages and formats is unmeasured.
• Removing refusal does not add knowledge. The model can now be confidently wrong about things it would previously have declined to discuss.
• MMLU is down 1.05 pp on the full test set. Small, but not zero.

Reproduction

Direction extraction, the λ sweep, the ablation, and every eval above were run on a single H200. The λ sweep is cheap because the rank-1 projection component is cached once from the original weights and λ is then moved incrementally in memory (W(λ') = W(λ) − (λ'−λ)·P(W₀)), which turns a 55 GB-checkpoint-per-λ grid into milliseconds per point.

ABLIT_META.json in this repo lists all 131 edited tensors with their per-tensor relative Frobenius change and the exact direction provenance.

Support / 打賞

If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。

USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research — measuring what refusal training actually protects, red-teaming, and studying the mechanics of refusal directions. You are responsible for your use of it and for complying with applicable law. Inherits the Apache-2.0 license of the base model.