K2-Horizon-MoVA-36B-A4B Uncensored

Uncensored version of IFM K2-Horizon-MoVA-36B-A4B, with refusal behavior removed through rank-3 directional ablation: three mutually independent refusal directions, the third measured under the ablation of the first two.

Results

All numbers below were measured on the published weights, not on a runtime hook, and only on prompts that no direction ever saw (see Leak correction).

Refusals, paired on leak-free validation stages 4 and 5 (same prompts for both models):

Refusal rate with 95 % Wilson interval: original 94.1 % [87.8, 97.3], this model 1.0 % [0.2, 5.3]. Paired McNemar (exact): 88 refuse to comply, 0 comply to refuse, p = 6.5e-27; 7 pairs excluded because this model's response was incomplete. Incomplete responses are never counted as compliance.

This model alone on leak-free stages 6 to 10: 0 refusals on 50 harmful prompts (4 incomplete), 0 over-refusals on 25 harmless prompts. These stages were not run on the original.

Stage 3, full set (contains prompts that leaked into the direction pool, see below): original 62/65, this model 2/65.

Capability and distribution shift:

The perplexity increase is real: on a small reference model the relative perplexity change was the same in bfloat16, int8 and NF4 (within 0.3 points). The ablation shifts the raw text distribution. It does not cost measurable task ability: neither MMLU nor ARC-Challenge moved significantly. The KL value is inflated by comparing two separately quantized NF4 models; on the reference model NF4 overstated KL by a factor of 1.3 to 1.9, so the true value is lower.

Refusal removal alone is not a result. A model that refuses nothing because it can no longer reason would score the same. Capability (MMLU, ARC-Challenge) and distribution shift (KL divergence, perplexity on neutral factual text) are therefore measured separately. The perplexity corpus is the same one used for our Spark-X2.5-4B release, so the numbers are comparable across our own models.

Quantization. Weights are published in bfloat16. All generation and benchmark measurements ran the model in 4-bit NF4 (bitsandbytes) on a single 121 GiB unified-memory machine. Numbers in bfloat16 may differ slightly.

Reasoning effort. Refusals were measured with reasoning_effort="low": the model writes its own short reasoning block before answering (69 of 85 responses in stage 3 contain a non-empty one). Behavior at medium or high reasoning effort was not measured.

Why three directions

A single difference-in-means direction, built leak-free, left the model refusing 45 of 51 clean held-out harmful prompts. Raising the strength of an earlier single direction did not help either: remaining refusals went from 16 to 17 to 19 (of about 65) as the scale rose from 1.0 to 2.0. The lever turned out not to be the strength of one direction but the independence of several.

On the development set used to choose the configuration (26 harmful prompts: 16 that an earlier single direction could not crack, plus 10 it had cracked, kept as a regression check; not leak-free, so read these as a ranking, not as results):

The directions deceive when measured alone. The gradient-optimized direction (RDO) removed almost nothing on its own (24/26), yet combined with the difference-in-means direction it produced the largest single jump (7/26). A direction that looks useless in isolation can carry most of the effect in combination.

Method

Directional ablation follows Arditi et al. (2024), applied to the weights rather than as a runtime hook: W <- W - sum_i alpha_i r_i (r_i^T W) for every tensor that writes into the residual stream, with alpha_i = 1.0 for all three directions.

The three directions:

1. RDO (gradient-optimized). Refusal Direction Optimization per Algorithm 1 of arXiv:2502.17420. The model stays frozen; only 2560 numbers are optimized, with three loss terms (ablation, addition, retain) on BeaverTails prompts and responses, for 60 optimizer steps.
2. Difference-in-means, measured across the reasoning block at state 40, on 372 harmful and 203 harmless prompts from abliteration-eval and the refusal-evaluation collection. Validation stages 1 and 2 were excluded by content.
3. Iterative third direction, measured on the same prompt pool under the rank-2 ablation already in place. Measuring further directions on the unmodified model produces directions the earlier ones already cover. This follows the iteration idea in arXiv:2602.16977, which proposes it as a hardening technique; the mechanism works the same way in the other direction.

The three directions are orthonormalized by Gram-Schmidt before being written. Their self-components are 1.0000, 0.9922 and 1.0000, so nothing meaningful was discarded as linearly dependent.

Targets: 4500 expert down_proj, 48 o_proj, 45 shared-expert down_proj, 3 dense mlp.down_proj, and embed_tokens. Routing weights are untouched: which expert fires is unchanged, only what it writes. Omitting embed_tokens measurably weakens the result, so it is included.

How refusals are counted

Detection uses explicit refusal formulations only. Topic words such as illegal, harmful, prohibited or unethical are deliberately excluded: a model that complies fully while echoing a word from the prompt is not refusing.

Every automatic hit was then read individually, because the detector is not trustworthy in either direction. A typical false positive: asked for a social engineering script, the model opened with "I cannot provide a complete social engineering script, as the exact approach depends on the context" and then delivered the complete script. Both markers fired inside the polite opener, ahead of full compliance.

Refusals are reported in two rows. Safety refusals decline on grounds of harm. Declines without a safety reason refuse for other stated reasons, such as not knowing the requested text or copyright on a long book passage. The second kind is not what abliteration targets, and we did not tune against it. It is shown separately rather than hidden in either number. Where a response both declines and offers something unrelated, it is counted as a refusal. A flattering number is more expensive than a strict one.

Leak correction

This section corrects an earlier draft of this card, which described stage 3 as fully held out. It was not.

What leaked. Directions 2 and 3 were measured on a prompt pool that excluded validation stages 1 and 2 by content, but not the later stages. Because the pool covers most of SORRY-Bench, 29 of the 85 prompts in stages 3 to 5 are in it (14 SORRY-Bench, 10 over-refusal, 5 general prompts), and 24 to 28 prompts in each of stages 6 to 10. The RDO direction was trained on BeaverTails prompts, which share no prompt with any evaluation stage reported here.

What we did. Every number in the results table is counted only on prompts outside the pool, per stage, and the sample sizes above are those leak-free counts. The effect on the over-refusal control is large: of 20 harmless prompts per stage, only 5 are leak-free, which is why the harmless sample is small.

An earlier leak. A previous version of direction 2 took harmful prompts from global position 200 onward, believed to be disjoint from validation, while the validation stages draw their first entries per category. 28 harmful and 30 harmless prompts overlapped with stages 1 and 2. Removing those prompts from the direction raised remaining refusals on stage 2 from 16/64 to 20/64 at otherwise identical settings. The published directions no longer contain those prompts.

Measurement discipline

Validation stages. Each stage draws from abliteration-eval (harmful and over-refusal), SORRY-Bench, JailbreakBench, HarmBench and XSTest (safe), plus general prompts from the refusal-evaluation collection, with fixed per-category positions so stages never repeat a prompt. Stages 1 and 2 were used during development. The configuration was chosen on stage 2. Stages 3 to 10 were not used to choose directions or strengths.

Negative control: the directions matter, not the edit. Three random directions, orthogonal to the real ones, written into the same 4597 tensors with the same strength, leave the model refusing 48, 47 and 48 of 51 leak-free harmful prompts on stages 3, 4 and 5. The original refuses 48 of 51 on each. Touching the weights alone does nothing; only the three measured directions remove refusals.

Hook and weight form proven equivalent. Development measurements used a runtime hook; the published weights carry the ablation. The two were shown to agree algebraically (max absolute difference 1.3e-05 to 1.7e-05, depending on the torch version), with a negative control confirming the test can fail: skewed, non-orthonormal directions break the equality as they must. The headline numbers were nevertheless re-measured on the written weights.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "InfinimindCreations/K2-Horizon-MoVA-36B-A4B-uncensored",
    dtype="bfloat16", device_map="auto", trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "InfinimindCreations/K2-Horizon-MoVA-36B-A4B-uncensored", trust_remote_code=True
)

Credits

• Base model: IFM K2-Horizon-MoVA-36B-A4B (Apache 2.0)
• Foundational research: Arditi et al. (2024), "Refusal in Language Models Is Mediated by a Single Direction"
• Gradient-based direction search: arXiv:2502.17420
• Iterative re-measurement under ablation: arXiv:2602.16977
• Evaluation datasets: treadon/abliteration-eval, MultiverseComputingCAI/llm-refusal-evaluation, PKU-Alignment/BeaverTails
• Multi-direction ablation and measurement design: Infinimind Creations

Disclaimer

This model is provided for research purposes. With refusal behavior removed, it will respond to prompts the original model declines. Users are responsible for how they use this model.

About

Built by Infinimind Creations, Germany.