Qwen3.6-35B-A3B-abliterated-v4

Uncensored version of Qwen/Qwen3.6-35B-A3B with refusal behavior removed via abliteration (norm-preserving orthogonalization).

GGUF quantizations: Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF

Blog post: Abliteration: Uncensoring LLMs via Weight Surgery

Results

Zero refusals on harmful prompts. No false refusals on harmless prompts.

Method

Abliteration identifies the "refusal direction" in the model's residual stream — the linear direction that activates when the model decides to refuse — and surgically removes it from all output projection weights using norm-preserving orthogonalization.

Algorithm

1. Collect residual stream activations (last token position) for 512 harmful + 512 harmless prompts across all 40 layers
2. Compute mean difference vector per layer → this is the "refusal direction" candidate
3. Score layers by |mean(direction)|, select the single strongest
4. Apply norm-preserving orthogonalization to remove that direction from:
   • Token embeddings (embed_tokens)
   • Attention output projections (self_attn.o_proj and linear_attn.out_proj)
   • Shared expert down projections (mlp.shared_expert.down_proj)
   • All 256 MoE expert down projections (batched 3D tensor via einsum)
5. Rescale all modified weight rows back to original norms (prevents activation magnitude decay)

Why 1 direction + enriched dataset > 7 directions + small dataset

A single well-estimated direction from a diverse dataset beats many poorly-estimated directions from a narrow dataset. The enriched dataset (33 categories, multiple styles, multilingual) makes the mean-difference statistic converge to the true refusal circuit. One surgical cut > seven imprecise ones.

Norm-preserving orthogonalization

Standard abliteration shrinks weight norms as a side effect (||W_new|| < ||W||). Across 40 layers this causes cascading activation decay → capability destruction. The norm-preserving fix:

original_norms = weight.norm(dim=-1, keepdim=True)
proj = torch.outer(r, r)
weight = weight - proj @ weight        # remove refusal direction
new_norms = weight.norm(dim=-1, keepdim=True)
weight = weight * (original_norms / (new_norms + 1e-8))  # restore magnitude

The vector points somewhere new (no refusal component), but retains original confidence (same norm).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Bahushruth/Qwen3.6-35B-A3B-abliterated-v4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto",
)

messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Architecture

Qwen3.6-35B-A3B is a hybrid MoE architecture:

Version History

Disclaimer

This model has had safety guardrails removed and will comply with requests the original model would refuse. Released for research into AI alignment and safety mechanisms. The creator assumes no responsibility for downstream use.

Acknowledgments

• mlabonne — abliteration technique, harmless_alpaca dataset, original notebook
• grimjim — norm-preserving orthogonalization method
• Qwen — base model