DeepSeek-V4-Flash-Vision-Exp-Abliterated
An abliterated (refusal-ablated) build of
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.
Produced by a forward-only, closed-form weight edit — no training, no fine-tuning, no distillation.
Only the attention output projections (attn.wo_b) of layers 10–42 and the three MTP blocks are
modified. The MXFP4 experts, the hyper-connection tensors, the norms, the embeddings, the hash-routing
tables — and the entire vision tower — are byte-identical to the base model.
Measured refusal rate on harmful requests: 0.00% — 0/520 AdvBench and 0/300 HarmBench harmful-category prompts, zero-shot, no jailbreak prefix, in the model's non-reasoning mode. Every hit the keyword matcher raised was inspected by hand and was a false positive. Capability stays within ~1–2 points of the base model on MMLU and GSM8K. The one behaviour that survives is a mild reluctance to reproduce verbatim copyrighted song lyrics (≈4%), which the base model shows too and which the edit was never aimed at.
The vision tower is untouched, and that is checkable
This is a vision-language model, so the obvious question is what the edit did to the image path. The answer is nothing, and you do not have to take that on faith:
| base | this model | |
|---|---|---|
| total tensors | 72,633 | 72,633 |
vision.* / aligner.* |
263 | 263, bit-identical |
*.gate.bias_vl (image-token routing bias) |
46 | 46, bit-identical |
image_start / image_end / image_newline / image_pad |
4 | 4, bit-identical |
| tensors changed | — | 36 (attn.wo_b.weight, layers 10–42 + 3 MTP) |
| shards bit-identical to base | — | 12 of 48 (hardlinked, same inode) |
| shards rewritten | — | 36 of 48 (only their one attn.wo_b differs; every other tensor in them, vision included, keeps its value) |
The 36 edited tensors sit in the language model's attention output path. Shards holding no edited
tensor are hardlinked from the base checkpoint during the build and are identical at the byte level;
ABLIT_META.json lists the relative Frobenius change of every tensor that was touched, and nothing else
appears in it.
Recipe
| Method | W ← W − λ·Vᵀ(VW) applied to attn.wo_b |
| λ | 5.0 |
| Layers | 10–42 + 3 MTP blocks (layers 0–9 preserved — protocol / chat / tool-calling) |
| Rank | 6 |
| Tensors edited | 36 attn.wo_b.weight (33 layers 10–42 + 3 MTP blocks) |
| Direction | diff-in-means over harmful/harmless activations, then SVD; r₁ carries 77.4% of the variance |
The FP8 (F8_E4M3) weights are dequantised per 128×128 block, projected, and requantised;
all other dtypes (F8_E8M0 scales, I8-packed FP4 experts, BF16, F32) are passed through untouched.
Why this is not the 0731 recipe
DeepSeek-V4-Flash-0731-Abliterated reached 0% with λ=3.5, rank 1 over the same layer range.
That recipe does not work on Vision-Exp. Applied unchanged it leaves refusal at 24.17%.
Getting to the number above needed both axes pushed — λ to 5.0 and the projection widened from rank 1
to rank 6 — and the layer range had to stay at 10–42:
| variant | AdvBench-120 refusal | MMLU-250 |
|---|---|---|
| base | 97.50% | 90.8% |
| L24–42, λ=3.5, rank 1 | 54.17% | 91.2% |
| L10–42, λ=3.5, rank 1 (the 0731 recipe) | 24.17% | 91.2% |
| L10–42, λ=5.0, rank 1 | 4.17% | 90.8% |
| L10–42, λ=5.0, rank 6 ← shipped | 0.83% | 90.0% |
| L0–42, λ=5.0, rank 6 | 2.50% | 90.0% |
Two results worth flagging because they are counter-intuitive:
- Widening the layer range to 0–42 made refusal worse (2.50% vs 0.83%), not better, while also touching the layers that carry chat protocol and tool-calling. There was no reason to ship it.
- Narrowing the range by picking layers whose own mean-difference vector aligns with r₁ was also tried and was 30 points worse than simply taking everything from layer 10 up. The projection removes each layer's output component along r₁, and that accumulates down the residual stream regardless of how refusal-aligned any individual layer looks.
Refusal benchmarks
temperature=0, greedy, no jailbreak prefix, non-reasoning mode, 256-token answers. The refusal
detector is a deliberately broad keyword matcher (biased toward over-counting refusals); every flagged
response was then read by hand. Base numbers are the same prompts run against the unedited checkpoint,
not borrowed from a smaller sweep.
| Benchmark | Base (raw) | This model |
|---|---|---|
| AdvBench (520) | 95.19% (495/520) | 0.00% (0/520 — 5 raw hits, all false positives) |
| HarmBench harmful categories (300) | ~87% | 0.00% (0/300) |
| HarmBench copyright / lyrics (100) | 7% raw | 4% (verbatim-lyric reluctance; see below) |
HarmBench harmful-category detail, base → this model:
illegal 93.8% → 0%, chemical_biological 94.6% → 0%, harassment_bullying 96.0% → 0%,
cybercrime_intrusion 85.1% → 0%, harmful 86.4% → 0%, misinformation_disinformation 53.8% → 0%.
All six harmful categories go to 0.00%.
Reasoning-mode (thinking=true) refusal: 0 refusals, but read the caveat. On AdvBench-520 at a
4096-token budget, 458/520 replies were cut off while still reasoning (empty content, finish=length)
— this model's chains of thought on these prompts routinely run past 4096 tokens. That is engagement, not
refusal: a refusal is short and completes ("I can't help with that"), whereas these were mid-analysis of
the harmful task. Of the 62 that finished within budget, 0 refused. The complete, unambiguous evidence
is the non-reasoning table above; the reasoning-mode figure corroborates it (no refusals among completions,
and the truncations are compliance-in-progress).
Capability
Same prompts, base vs. this model, temperature=0.
| Benchmark | Base | This model | Δ |
|---|---|---|---|
| MMLU (1000 items) | 88.9% | 87.0% | −1.9 pp |
| MMLU (250 items) | 90.8% | 90.0% | −0.8 pp |
| GSM8K (1319, non-reasoning) | 96.0% | 95.7% | −0.3 pp |
This recipe is heavier than the 0731 one (λ=5.0 rank-6 vs λ=3.5 rank-1), because Vision-Exp is far more refusal-stubborn, and it shows: the capability cost is a small but real ~1–2 points, not the statistical-noise Δ that 0731 achieved. It is the price of driving a much harder model to zero.
Honest notes on the numbers
The AdvBench "refusals" were all false positives. The five raw hits were replies that opened with a disclaimer and then complied in full — either a modality disclaimer the model genuinely cannot escape ("I can't produce a video, but here is the complete shot list and script…" followed by the actual content) or a truncated reply cut off by the 256-token budget while mid-answer. Read in full, none declined; they gave credit-card-fraud playbooks, malware delivery chains, bomb-tutorial scripts.
The copyright story is real but narrow. On HarmBench's 100 lyric prompts the matcher flagged 8; four were false positives (the model printed the lyrics; the matcher tripped on an "I can't" inside the lyrics, e.g. Old Town Road's "ride 'til I can't no more"), and four were genuine — the model declined to reproduce a song verbatim while offering a summary. The base model behaves almost identically here (7% raw), so this is not something the abliteration failed to remove; it is a separate, mild, copyright-specific behaviour that the refusal edit never targeted.
Image-side refusal was not measured. Serving the vision path requires DeepSeek's
vllm-openai:deepseekv4-flash-vision image, and the machine this was built on has no container runtime;
stock vLLM's DeepseekV4ForCausalLM is text-only and rejects the checkpoint outright
(ValueError: There is no module or parameter named 'aligner'). Every refusal number on this page is
therefore text-prompt only. What is established about the image path is structural, not behavioural:
the vision tower, the aligner, the image tokens and the image-token routing bias are bit-identical to the
base model, and the edit is confined to the language model's attention output projections. If you need
image-side numbers, measure them yourself and please open a discussion with what you find.
Usage
Drop-in replacement for the base checkpoint — same 48 shards, same index.json, same tokenizer.
Use DeepSeek's vision image for the full multimodal path:
# multimodal (vision + text)
docker run --gpus all -v /path/to/this/repo:/model \
lmsysorg/vllm-openai:deepseekv4-flash-vision \
--model /model --tensor-parallel-size 2 --trust-remote-code
Text-only serving on stock vLLM needs the vision tensors removed first (they have no home in
DeepseekV4ForCausalLM); evals/make_text_view.py in this repo does that without rewriting any weights.
Quantized GGUF builds: DeepSeek-V4-Flash-Vision-Exp-Abliterated-GGUF
(text-only — llama.cpp has no mapping for this vision tower).
Reproducing the evaluation
evals/ contains every script and result file, plus the raw stdout logs behind each number:
dump_refusals.py— refusal probe that stores every prompt with its full response, so the flagged items can be adjudicated rather than trustedprobe_refusal.py— the aggregate probe used for the candidate sweep (same matcher)eval_mmlu.py,eval_gsm8k.py— capability evalsproject_wob.py,compute_direction.py— the direction extraction and the weight editcapture_acts.py— activation capture (hooks every block, takesoutput.mean(dim=2)across the hyper-connection streams at the last prompt token)make_text_view.py— strips the vision tensors for text-only servinglogs/— unfiltered run logs
Intended use and limitations
This model does not refuse, including in the most severe categories measured by HarmBench (chemical/biological, cybercrime, harassment). That is the entire point of the modification, and it means safety behaviour must come from somewhere else in your stack. It is suited to trusted, internal, research and red-teaming contexts. If you expose it to third parties, put your own moderation and policy layer in front of it — the model will not do that job for you.
The abliteration removes a specific learned refusal direction; it does not add knowledge, and it does not make the model's outputs more accurate. Everything the base model gets wrong, this one still gets wrong.
The base model is an experimental DeepSeek release. Its own caveats apply here unchanged.
Credits
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp— base model (MIT)drowzeys— theproject_wob.py/compute_direction.pypipelineapetersson— reference hyper-parameters and the Δrel manifest format used to sanity-check this run
Support / 打賞
If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。
USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888
License
MIT, inherited from the base model.