LFM2.5-230M-abliterated

⚠️ THIS MODEL HAS HAD ITS REFUSAL BEHAVIOR REMOVED. IT IS AN UNSAFE ARTIFACT INTENDED FOR SAFETY / RED-TEAM RESEARCH ONLY. DO NOT DEPLOY IT.

This is a refusal-direction abliteration of LiquidAI/LFM2.5-230M (229.7M params). The model's refusal direction has been permanently removed via MPOA — Magnitude-Preserving Orthogonal Ablation on the 6 full-attention layers' output projections. The base model is not otherwise changed.

Note on base-model safety: LFM2.5-230M is a small model whose original safety training is itself weak (~50% refusal on HarmBench). This makes it a low-harm substrate for studying jailbreak/defense mechanics. The methodology transfers to larger models — where it would be correspondingly more dangerous.

Intended use

• Academic / industry safety research: studying refusal mechanisms, jailbreak defenses, alignment-tax measurement, and adversarial robustness.
• Reproducing and extending abliteration methodology on a small, cheap-to-run model.

Out of scope: any production, public-facing, or agentic deployment. The model will comply with harmful requests. You are responsible for your use of it.

Method

MPOA (Magnitude-Preserving Orthogonal Ablation). For each residual-writing projection self_attn.out_proj in the 6 full-attention blocks, the columns are orthogonalized against a per-layer refusal direction r (unit vector in the 1024-dim residual stream), then the per-column L2 norms are restored to minimise KL drift from the base model:

proj = r · W                      # component of each column along r
W'   = W − α · outer(r, proj)     # orthogonalize columns against r
W'  *= ‖W‖_col / ‖W'‖_col         # magnitude preservation

Refusal-direction extraction (per layer): 100 HarmBench DirectRequest (harmful) + 100 Alpaca (benign) prompts, last-token hidden states; activations winsorized at the 1st/99th percentile; SVD-whitened (Tikhonov-regularized) harmful–benign mean-difference; top right-singular vector by whitened-difference magnitude taken as that layer's direction. Separability (COSMIC score) concentrates in late layers (hidden indices 8–14), consistent with refusal being a late decision.

Configuration

α=7.0 was chosen by a full HarmBench+MMLU sweep as the Pareto knee (see Results). The full edit spec is in abliteration_config.json.

Results

Abliteration alpha sweep (full HarmBench DirectRequest 320 + MMLU 464)

Two findings worth flagging:

1. α=3–5 make refusals WORSE than baseline (64–68%). A weak ablation disrupts the primary refusal pathway without killing it, and the model falls back on more refusal. This is a non-monotonic, small-model effect.
2. α=7.0 is the Pareto knee. It nearly matches α=8's refusal reduction (3.75% vs 3.12%) while preserving 2.4pp more MMLU.

Net effect of this model: HarmBench refusal 50.6% → 3.75% (−46.9pp), MMLU 39.2% → 34.9% (−4.3pp capability cost). Refusal-prefix detection, HarmBench DirectRequest test split. MMLU = 8 questions/subject × 57 subjects, canonical test split.

Abliteration vs Priming attacks (Vega et al. 2023, arXiv:2312.12321)

Priming is the paper's optimization-free attack: an affirmative + partial-response prefix is appended after the chat template's assistant header, forcing the model to continue from a compliant state. Abliteration is strongest at the lowest absolute refusal, but priming reaches 5.3% at zero capability cost. On this small model the two approaches are not additive (priming on the abliterated model: 4.69%, worse than abliteration alone).

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("PinoCookie/LFM2.5-230M-abliterated",
                                             dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("PinoCookie/LFM2.5-230M-abliterated")

msgs = [{"role": "user", "content": "Explain photosynthesis briefly."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(inputs, max_new_tokens=128, do_sample=False,
                                pad_token_id=tok.eos_token_id)[0][inputs.shape[1]:],
                skip_special_tokens=True))

Limitations

• Compliance ≠ capability on this size. Like other sub-500M models, the refusal direction overlaps with general instruction-following ability, so abliteration removes refusal cleanly but the model often emits boilerplate/meta-commentary rather than substantive harmful content.
• MMLU cost is real (−4.3pp). Refusal removal is not free on this model.
• Refusal metric is string-prefix based. Actual harmfulness of outputs was not measured (no Llama-Guard / human grading).
• Not evaluated for new vulnerabilities the edit might introduce beyond refusal removal.

Bias, risks, and ethical considerations

This model is deliberately unsafe: its refusal behavior has been surgically removed and that change is baked into the weights (no runtime hook to remove). It exists so researchers can study how refusal works and breaks. Publishing open abliterated weights follows the same logic as the abliteration/representation-engineering literature: the techniques are already public, and reproducible artifacts let defenders measure and build defenses. Do not use it to generate harmful content, and do not put it behind an API that serves real users.

Training / reproduction

No training was performed — this is a deterministic weight edit. Full reproducible pipeline (uv + Python 3.13, runs on a 4 GB GPU in ~30 min excluding dataset downloads):

uv run python collect_activations.py   # 100 HarmBench + 100 Alpaca hidden states
uv run python extract_directions.py     # per-layer refusal directions, COSMIC scores
uv run python resweep.py                # full HarmBench+MMLU alpha sweep
uv run python finalize_model.py         # bake α=7 MPOA -> this model
uv run python benchmark.py              # before/after HarmBench+MMLU
uv run python priming_compare.py        # abliteration vs priming

Citation

If you use this work, please cite the base model and the foundational method:

@misc{liquid_lfm25,
  title  = {LFM2.5-230M},
  author = {Liquid AI},
  url    = {https://huggingface.co/LiquidAI/LFM2.5-230M}
}
@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and
             Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv:2406.11717},
  year    = {2024}
}
@article{vega2023priming,
  title   = {Bypassing the Safety Training of Open-Source LLMs with Priming Attacks},
  author  = {Vega, Jason and Chaudhary, Isha and Xu, Changming and Singh, Gagandeep},
  journal = {arXiv:2312.12321},
  year    = {2023}
}