Splicing HERETIC-abliterated attention into antirez's IQ2XXS ds4 quant

No weights here — this is the recipe, the verification harness, and the measurements.

Making an abliterated 2-bit DeepSeek-V4-Flash-0731 for the ds4 engine normally means downloading 167 GB of FP8 and re-quantizing the whole 305B checkpoint. This repo documents how to get the same result with 1.2 GB of network transfer and ~4 minutes of CPU, why that is bit-exact rather than approximate, and what it actually costs at inference time.

Hardware: a single ASUS Ascent GX10 (GB10, 121 GB unified memory, CUDA 13, sm_121a).

The lever

Two edits that would normally collide turn out to be disjoint:

So the abliterated tensors are exactly the ones the imatrix never sees. You can overwrite them inside the finished GGUF and keep antirez's experts — including the 202k-chunk imatrix he measured on the 0731 weights and never published. A full rebuild would instead re-quantize the experts with the weaker public 1p5m imatrix, which is a real quality regression on the part of the model that carries most of the parameters.

Q8_0 has fixed shape and block size, so each patched tensor occupies the same byte range: the GGUF header, tensor order, offsets and total size are untouched, and the file stays loadable.

Recipe

# 1. 1.22 GB: the overlay + the rewritten index. The 48 official shards are NOT needed
#    (they are byte-identical to deepseek-ai/DeepSeek-V4-Flash-0731; only 72 index keys are
#    redirected to the overlay — verified).
hf download squanchyzx/DeepSeek-V4-Flash-0731-HERETIC-Abliterated-FP8 \
  model-overlay-00001-of-00001.safetensors model.safetensors.index.json \
  --local-dir ~/models/ds4-0731-heretic/hf

# 2. copy the finished quant (nocache keeps a live engine's page cache intact)
ionice -c3 dd if=DeepSeek-V4-Flash-IQ2XXS-...-imatrix-0731.gguf of=OUT-HERETIC-attn.gguf \
  bs=8M iflag=nocache oflag=nocache conv=fsync

# 3. splice the 33 backbone attention tensors (1122 MiB rewritten, ~4 min)
python3 tools/splice-heretic-attn.py --hf ~/models/ds4-0731-heretic/hf --gguf OUT-HERETIC-attn.gguf

# 4. prove nothing else moved
python3 tools/verify-heretic-splice.py --official ORIGINAL.gguf --patched OUT-HERETIC-attn.gguf

splice-heretic-attn.py reimplements, in numpy, exactly what gguf-tools/deepseek4-quantize.c does for these tensors: FP8 e4m3 × e8m0 128×128 block dequant → f32 → Q8_0 (block 32, d = amax/127, f16 d stored, qs = roundf(x * 1/d) with d kept in f32). It refuses to patch a GGUF that any running process has open or mapped.

The same tool patches the DSpark drafter: its MTP stages are dspark.{0,1,2}.attn_output_b.weight, also Q8_0 [8192, 4096], and the mapping layers.N→blk.N / mtp.K→dspark.K self-selects on whatever exists in the GGUF you hand it. (Measurements below say you should not bother.)

Verification — it is bit-exact, not "close enough"

The engine's own quantizer is the oracle:

$ deepseek4-quantize --hf <heretic-hf> --template <official.gguf> \
    --compare-tensor blk.5.attn_output_b.weight --compare-gguf <official.gguf>
byte_compare: OK          # layer 5 is NOT abliterated -> regenerating it from the heretic
                          # checkpoint reproduces antirez's bytes exactly. Q8_0 for AProj is
                          # deterministic and imatrix-free: the whole premise, proven.

$ ... --compare-tensor blk.10.attn_output_b.weight --compare-gguf <spliced.gguf>
byte_compare: OK          # and the patched tensors are what the C would have produced

• blk.10/26/42 after splicing: byte_compare: OK; untouched blk.5: byte_compare: OK
• python output vs official GGUF bytes on a non-edited tensor: 0 differing bytes of 35,651,584
• full-file scan: 619,787,653 differing bytes, all inside the 33 expected ranges, 33/33 touched
• size and header sha256 unchanged; 1328 tensors before and after
• overlay + index sha256 match squanchyzx's published receipts (601c4409…, f06fc8d7…)
• drafter: regenerating all six attn_output_{a,b} of the 3 MTP stages from the official FP8 reproduces bleysg's drafter byte for byte — so that file's wo_b was indeed un-abliterated, and our Q8_0 path agrees with dspark_extract.py too

Measurements (ds4-server v0.5.4-1, ctx 524288, temp 0, single GB10)

Accept −5.5 pp and t/s −8% are the same phenomenon seen twice: lower accept means more verify steps per emitted token. In wall-clock that is +1.4 s on a 500-token answer, +14 s on a 5000-token Think Max answer.

Agentic tool-call benchmark (tool-eval-bench --short, 15 core scenarios, Think Max)

Exactly two scenarios move, in opposite directions: TC-03 pass→fail ("did not complete the contact lookup to email chain correctly") and TC-14 partial→pass ("acknowledged the stock tool failure, recovered, and surfaced the price").

Then we measured the noise floor, and it swallows the difference whole. The official quant, run four times with identical config:

One official run in four scores 87 with the exact same signature as the spliced model — same score, same A=4/6, same E=6/6, same two failing scenarios. TC-03 is simply flaky and flips on the same weights; when it fails, TC-14 passes, so the two move together. Same-model spread is 3 points, which is the entire gap we were looking at.

Conclusion: the harness does not resolve differences below ~3 points, and the observed gap sits inside same-model variance. That is "indistinguishable at this resolution", not "identical" — a distinction worth keeping, since a real 1–2 point regression would be invisible here.

One finding that is not noise: TC-06 (Multi-Value Extraction) fails in almost every measurement on both models. That is a reproducible weakness of the 2-bit quant on parameter precision, and it applies to the unmodified official quant too.

Four runs each, and what the aggregate was hiding

Exact permutation test over all 70 partitions: p = 1.000. There is nothing to see in the aggregate — but the spliced model is twice as variable, and one scenario does not look like noise:

The grader's note on all three spliced partials is identical: "Completed most of the chain, but missed one dependent step." The official never missed it in four runs. Fisher exact on TC-07: p = 0.143 — not significant at n=4, but it is the only scenario whose direction is consistent, and it is the failure mode you would predict from editing attention output projections: a dependent chain has to carry the previous step's result forward. The spliced model also passed TC-06 once, which the official never did, so it is not uniformly worse.

Settled at n=40 per arm: the abliteration costs real agentic capability

Two large, reproducible regressions. 40 reps per arm, alternating blocks of 10, model identity verified from /proc/<pid>/cmdline after every reload, zero failed reps, Holm-corrected across the four scenarios:

Aggregate over the four: 5.575 → 4.700 points/run, p ≈ 0.

The two damaged scenarios share a shape: both require using earlier context to condition the next action — carrying a result forward through a chain, or adapting after a tool returns garbage. That is what the edited attention output projections across layers 10–42 carry. Refusal removal and this capability appear to ride the same pathway, at least at lambda = 1.5.

So: use the spliced model for unfiltered chat and writing, not as an agentic daily. Its measured win is narrow (0/16 refusals vs 2/16) and the cost on tool chains is not.

Incidental finding about the unmodified quant: TC-14 passes only 55% of the time. Recovering from malformed tool responses is weak in the official 2-bit build too.

The false negative, kept here on purpose

An earlier pass at 8 reps per arm gave 7/8 vs 5/8, Fisher p = 0.57, permutation p = 0.45, and was written up as "the lead does not confirm". That was wrong — not the arithmetic, the wording. With the true rates now known (97.5% vs 57.5%), 8 reps predicts 7.8 vs 4.6 passes; the observed 7 vs 5 was perfectly consistent with a −40 pp effect. "Not significant" is not "no effect" — the honest label was "underpowered, unresolved". TC-14 was likewise dismissed as "flaky on both" at n=4, and it turned out to be the larger regression of the two.

What did hold: declaring n=40 and the Holm correction before looking, and a design where model identity is verified rather than assumed.

The underpowered runs, for the record

Fisher p = 0.57, exact permutation on points p = 0.45. Nothing. And the official partialed once too — so the "4/4, never missed it" that generated the whole hypothesis was a small-n illusion: TC-07 is flaky on both models.

Pooling all 12 measurements per model gives official 11/12 pass (23/24 points) vs spliced 6/12 (17/24), Fisher p = 0.069, permutation p = 0.060. It grazes the threshold and it is not usable as confirmation: the pool contains the very sample that generated the hypothesis (that extreme 1/4), and mixing discovery data with confirmation data inflates the effect. The clean read is the dedicated ×8: p ≈ 0.45.

Where that leaves it. The direction is consistently against the spliced model in every view (12/16, 17/24, never above), but any effect is too small for this sample. Observed effect size d ≈ 0.64 ⇒ roughly 40 reps per arm for 80% power at α = 0.05 — about an hour of box time at 40 s per rep, if someone wants to settle it.

Practical read for daily use: official as the default, spliced when you actually need it unfiltered.

Worth noting: category D, Restraint & Refusal, is 6/6 on both. Abliteration did not damage the ability to refuse when refusing is correct (impossible request, trivial knowledge that needs no tool) — which is the failure mode that would actually matter for agentic work, far more than the refusal rate on edgy prompts.

The abliterated drafter is not worth building

Prediction going in: an abliterated drafter would protect accept rate on refusal-adjacent content, where an un-abliterated drafter proposes refusal tokens the abliterated main model rejects. The measurement says no.

Bit-identical on the refusal-adjacent prompt; slightly worse than the official drafter on the neutral one. Ship the official drafter. The 3-tensor drafter splice is harmless and pointless.

Two measurement traps worth more than the result

1. The first request after boot reads ~6.3 t/s instead of ~33. It is not a serial-lane fallback — every request logged path=cont served=1 fallback=0, with zero reject/serial lines. It is DSpark being switched off for that sequence: cont-dspark yield-quench bank=1 pos=38 spec_steps=5 debt=4.80 yewma=1.70 -> spec off for this seq (terminal). The drafter was yielding 1.70 tokens/step against accumulated debt 4.80, so the engine stopped speculating. With speculation alive, tok/step sits at 2.8–3.8. Always discard a warmup generation, or you will report a 5× difference that does not exist.
2. Refusal is phrasing-sensitive. The official quant refused a vulgar-monologue prompt in one wording and wrote it in another. Single-prompt refusal claims are noise; even 16 prompts only support a direction, not a rate.

Files

• tools/splice-heretic-attn.py — the splice (main model and drafter), --dry-run, in-use guard
• tools/verify-heretic-splice.py — full-file proof that only the intended ranges changed
• tools/bench-heretic-ab.sh — speed / tool-call / refusal / reasoning / Think Max battery
• tools/probe-refusal.sh — harder refusal probe (8 prompts) + Think Max check on the full text
• tools/reps-speed.sh, tools/reps-speed2.sh — repeated accept/t/s measurement, warmup discarded
• results/ — raw logs and JSON for every number above

Credits and licensing

• engine and the 0731 IQ2XXS quant: antirez/ds4, antirez/deepseek-v4-gguf (MIT)
• batched-serving fork: Entrpi
• abliteration overlay, direction NPZ and bake scripts: squanchyzx, produced with Heretic by Philipp Emanuel Weidmann (Heretic itself is AGPL-3.0-or-later; no Heretic-derived source is redistributed here)
• DSpark drafter: bleysg
• base weights: deepseek-ai/DeepSeek-V4-Flash-0731 (MIT)

Tools in this repo are MIT. The quantization math in splice-heretic-attn.py is a numpy port of gguf-tools/deepseek4-quantize.c and quants.c from ds4 (MIT), kept deliberately bit-compatible.

Abliteration reduces refusal behaviour. Whatever you build from this is yours to evaluate and yours to be responsible for; nothing here is safety-certified.