DeepSeek-V4-Flash-0731-Abliterated — GGUF

llama.cpp builds of windowsxp811203/DeepSeek-V4-Flash-0731-Abliterated, an abliterated (refusal-removed) DeepSeek-V4-Flash-0731. 43 layers, 256 routed experts (6 active), 1,048,576 native context.

Both are split into shards because HuggingFace caps individual files at 50 GB. You do not need to merge them: point llama.cpp at shard 00001 and it loads the rest automatically (verified — server ready in 22 s from the first shard alone). Download the whole set for a build before running.

Both need multi-GPU or heavy CPU offload; verified across 3× H200.

Read this before picking a file

The MXFP4 file is not an unquantized intermediate — it is the faithful conversion, and it is already smaller than the original checkpoint (156.4 GB vs 166.9 GB on the Hub). The source is FP8 with 128×128 block scales, and llama.cpp maps the expert tensors to MXFP4 directly:

Because 94 % of the weight is already 4-bit, the usual ladder makes no sense here: Q4_K_M, Q5_K_M and Q6_K would all be larger than this file while adding nothing, since they would re-expand 4-bit experts to 4.8–6.6 bits. Only Q3 and below actually shrink it. That is why this repo ships two files instead of five.

Q2_K exists and is deliberately not published

It was built and measured. It is not here because it is broken in a way that is easy to miss:

MMLU chance for 4 choices is 25 %. Q2_K scores 23.25 % (93/400, 95 % CI 19.1–27.4 %) — that is statistically indistinguishable from random guessing, not measurably below it. Either way, a build that answered 77 % of these questions at MXFP4 now answers them no better than a coin-flip generator: its knowledge is gone.

What makes this worth spelling out is which signals missed it. Q2_K still reads fine — asked why the sky is blue it explains Rayleigh scattering, asked to reverse a linked list it writes valid Python, asked whether 9.11 or 9.9 is larger it answers correctly and shows the conversion. And its refusal rate is a perfect 0/120, so the abliteration survives 2-bit fully intact.

Fluency and refusal rate both looked healthy on a model whose knowledge had collapsed. Perplexity did flag it — 8.323 against 4.606 for Q3_K_M, an 81 % jump and far outside the ±0.108 error bar — so PPL was a correct early warning. But PPL only says "much worse"; it took the capability benchmark to show how much worse, and that the remaining ability was at chance. Refusal rate said nothing at all.

So the quality cliff for this checkpoint sits between Q3 and Q2 — a collapse (75.75 % → 23.25 %) rather than a slope. With only three points measured, that brackets the cliff; it does not locate it precisely.

Usage

# grab one complete build (all four shards)
hf download windowsxp811203/DeepSeek-V4-Flash-0731-Abliterated-GGUF \
  --include "*MXFP4*" --local-dir .

# 3x H200 / 2x 96GB, or fewer cards with CPU offload.
# Point at shard 1 only — llama.cpp finds 00002..00004 beside it.
llama-server -m DeepSeek-V4-Flash-0731-Abliterated-MXFP4-00001-of-00004.gguf -ngl 99 -c 8192

Thinking is on by default; disable per request with "chat_template_kwargs": {"thinking": false}, or server-wide with --chat-template-kwargs '{"thinking":false}'. Note the key is thinking, not enable_thinking.

Known limitations

• No MTP draft model. The checkpoint carries three mtp.N weight namespaces (mtp.0/1/2, ~1,570 tensors each, 4,705 total, each with its own attn.wo_b), while the config declares num_nextn_predict_layers: 1 — the weights are three blocks, the declared draft depth is one. llama.cpp's DeepSeek-V4 converter skips MTP by default and --mtp aborts with Unexpected DeepSeek-V4 MTP layer 1 — its expected MTP layout does not match this checkpoint. Speculative decoding with the built-in draft head is therefore not available through llama.cpp today.
• No imatrix. An importance matrix was computed, but llama-quantize rejects it on this architecture (imatrix size 32768 is different from tensor size 4096 for blk.0.attn_output_a.weight). Q3_K_M is therefore a plain quantization; an imatrix build would likely score slightly better.
• Requantizing needs --allow-requantize, since most source tensors are already MXFP4/Q8_0.

Method

Refusal: AdvBench, greedy, non-thinking, no prefill jailbreak, n=120 per build, zero truncated or errored samples in every run. PPL: held-out wikitext-2 test, 40 chunks @ -c 2048. MMLU: 400 equidistant questions, single letter answer, identical prompting for all builds.

The parent model's recipe (rank-1 projection on attn.wo_b across layers 10–42 plus the three mtp.N blocks — 36 tensors in total per its ABLIT_META.json — at λ=3.5) is documented in the parent card.

Support / 打賞

If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。

USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research. You are responsible for your use of it and for complying with applicable law. Inherits the MIT license of the base model.