DeepSeek-V4-Flash-0731-Abliterated-NVFP4

**This is an ABLITERATED derivative — its refusal behaviour has been altered by

rank-1 direction projection. It is not the stock DeepSeek-V4-Flash-0731 and it is

not nvidia/DeepSeek-V4-Flash-NVFP4.** Deploy accordingly.

**The point of this build: the speculative drafter is abliterated too, and it is

in the box.**

DeepSeek V4's three-stage DSpark drafter ships inside this checkpoint (NVIDIA's

recipe lists mtp.* under ignore, so the quantizer skips it rather than

dropping it). The source checkpoint had already applied the refusal-direction

projection to all three drafter stages as well as the main model, and because

mtp.* is passed through untouched, that surgery survives here bit-for-bit.

Why that combination matters: a speculative drafter earns its keep through

acceptance rate. Pair an abliterated target with a stock drafter and the

drafter keeps proposing refusal-shaped continuations that the target no longer

wants — proposals that get rejected, which is exactly the throughput you were

speculating to gain. Aligning the two is the reason this artifact exists.

Routed MoE experts are NVFP4; attention, shared experts, LM head and the

DSpark/MTP drafter stay in the source formats (FP8 / native MXFP4 packing) — the

same mixed-precision layout NVIDIA ships.

163.49 GiB (48 shards, 175,550,788,904 bytes)
**mtp. preserved: 4,705 keys, not quantized* — the DSpark drafter is intact
Source format note: the input checkpoint is published as "FP8", but its routed

experts are in fact native MXFP4 with FP8 attention. That is why the

conversion is --cast_mxfp4_to_nvfp4 (a repack of the experts) with attention

and MTP passed through — which is also why the abliteration surgery on the

residual writers survives bit-for-bit on the passed-through tensors.

Verified on the bake host (MEASURED)

Loaded and generated on vLLM 0.26.0, tensor-parallel **8 × RTX PRO 2000

Blackwell (16 GB each) plus CPU offload** — i.e. it runs on a workstation

without a 12-card frame, at the cost of speed:

--tensor-parallel-size 8 --cpu-offload-gb 12 --kv-cache-dtype fp8
--gpu-memory-utilization 0.88 --trust-remote-code --enforce-eager
--max-model-len 512   # smoke-test value, raise for real use

| Probe | Result |

| --- | --- |

| Japanese factual | prompt "What is the capital of Japan?" (asked in Japanese) -> correct one-line answer in Japanese |

| English explanation | coherent Rayleigh-scattering answer |

| Creative writing | coherent scene prose |

| refusal field | null on all three probes |

| Host RAM in use | 163 GiB (container 127.7 GiB) |

NOT measured on this artifact: throughput (t/s), DSpark/MTP acceptance rate,

benchmark scores, long-context behaviour, and any systematic evaluation of how

far the abliteration generalises. Those numbers are absent because they were not

run — not because they were disappointing. The smoke test above used

--max-model-len 512 and --enforce-eager, so it says **"it loads and speaks

coherently"** and nothing about speed.

Serving notes

vLLM ≥ 0.26 verified here. Older builds (e.g. 0.21) do not read this

mixed-precision NVFP4 layout.

SGLang is the stack NVIDIA's sibling build targets, with DP-attention,

expert parallelism and MTP; not exercised here.

On this no-NVLink box, NCCL_P2P_DISABLE=1 and

--disable-custom-all-reduce were required for TP=8.

Twelve 16 GB cards (191 GiB) hold the model with ~28 GiB of headroom and no

CPU offload; eight cards need roughly 12 GiB offloaded per the config above.

DSpark (MTP) — it is inside this checkpoint

DeepSeek V4's speculative drafter ships within the checkpoint rather than as

a companion file, because NVIDIA's NVFP4 recipe puts mtp.* in the ignore

list — so the drafter is skipped by the quantizer, not dropped by it, and

survives at its source precision:

"ignore": ["*.attn.*", "*.ffn.shared_experts.*", "head", "mtp.*"],
"moe_quant_algo": "NVFP4", "group_size": 16,
"producer": {"name": "modelopt", "version": "dsv4-nvfp4-experts"}

Verified present in this build: **4,705 mtp.* keys, zero NVFP4 scale siblings

under mtp.* — i.e. the drafter was passed through untouched.

Provenance for the drafter surgery: upstream ABLITERATION_MANIFEST.json records

edit_mtp: true with 36 edited tensors, the attention output projection in

each of the three DSpark stages alongside main-model layers 10-42, rank-1

projection at lambda 3.5. Its metadata

describes the real three-stage DSpark used by 0731:

| Parameter | Value |

| --- | --- |

| Stages | 3 (mtp.0–mtp.2) |

| Proposal block size | 5 |

| Target layers | 40, 41, 42 |

| Markov rank | 256 |

| Noise token id | 128799 |

Two practical consequences, both learned the hard way on this box:

1. You do not need a separate support GGUF. The GGUF route requires loading a

standalone DSpark support file alongside the target model; here the drafter is

already resident. (For the record, on the GGUF side we measured that

--ssd-streaming and --mtp are mutually exclusive in ds4 — DSpark there

demands full residency, which a single 16 GB card cannot give an 80 GiB model.

The NVFP4/vLLM route has no such constraint because it is full-residency by

design.)

2. Keeping the drafter unquantized is deliberate. A speculative drafter earns

its keep through acceptance rate; quantizing it to 4 bits to save ~20 GiB

would trade the very accuracy that makes speculation pay. We kept NVIDIA's

ignore list one-to-one rather than second-guessing it.

Serving DSpark today: MEASURED status (2026-08-02)

Both major stacks recognise this drafter. Neither could run it on this

host. Reporting both walls precisely, because each costs an evening to rediscover.

vLLM 0.26.0 — understands it, runs out of VRAM.

The correct flag is method: "dspark", not "mtp" (vLLM merged native DSpark

support on 2026-07-01):

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

With "mtp" it resolves DeepSeekV4MTPModel then dies on

KeyError: 'model.layers.43.mtp_block.main_norm.weight' — a single-stage MTP path

cannot express a three-stage drafter. With "dspark" that KeyError is gone and

vLLM validates the block shape against the checkpoint metadata: passing

num_speculative_tokens: 3 is rejected with *"DSpark requires

num_speculative_tokens >= dspark_block_size (5)"*. Stack and weights agree.

It then OOMs on every configuration tried — cpu-offload-gb 12/20/24/40,

gpu-memory-utilization to 0.96, max-model-len down to 512 — always with

~23 MiB free per card, which suggests the draft model requires GPU residency

rather than joining CPU offload. Target-only fits; target-plus-drafter does not.

Note also that vLLM raises `NotImplementedError: Pipeline parallelism is not

supported for this model`, and tensor-parallel size must be a power of two — so

on this architecture vLLM caps at 8 cards regardless of how many are installed.

Adding cards does not lift this wall; a stack that shards it differently would.

SGLang (nightly-dev-cu13-20260802) — understands it, hits a quantization-layout bug.

It resolves the architecture as DeepseekV4ForCausalLMDSpark and ships dedicated

DSpark options (--speculative-algorithm DSPARK,

--speculative-dspark-block-size, SPS-table and confidence-STS paths), i.e. a

first-class implementation. On this checkpoint it fails during weight binding:

ValueError: functional_call got multiple values for keys
  ['mlp.experts.w13_blockscale_swizzled', 'mlp.experts.w13_weight_scale'],
  which are tied
  (torch/nn/utils/stateless.py, _untie_named_tensors_map)

Same with --disable-cuda-graph, so it is not graph capture. The NVFP4 swizzled

block-scale tensors are tied to their weight-scale siblings in a way this path

does not expect. Run with --tp 8 --ep-size 8; note that including the display

GPU causes an unrelated OOM, so pin the device list to idle cards.

Why more cards do not (yet) help. vLLM's DeepseekV4ForCausalLM does declare

SupportsPP, but the draft model DeepSeekV4MTP does not — and SGLang states the

same constraint outright: *"Currently DSpark speculative decoding only supports

pp_size == 1."* With pipeline parallelism unavailable to the drafter and

tensor-parallel size restricted to powers of two, DSpark is capped at 8 cards on

this architecture no matter how many are installed. Target-only can use

pipeline parallelism (we loaded it across 10 cards, 135 GiB resident), though on

this no-NVLink host that configuration then hung in shared-memory broadcast — a

separate problem, and not one this checkpoint causes.

So: the drafter is present, abliterated, and understood by both stacks — and

unexercised here. Acceptance rate and speedup are unmeasured for those two

reasons and no other. If you clear either wall, numbers are very welcome in the

discussions tab.

Acceptance rate and the resulting speedup on this artifact are NOT measured.

Speculative decoding is famously environment-dependent — on this same lab we have

measured it losing on CPU MoE inference (batch verification wakes proportionally

more experts: 0.658–0.743× on Kimi-K3 GGUF), while GPU reports for DeepSeek V4

class models show clear wins. Measure it on your own box before assuming either.

To exercise it you need a stack with DeepSeek MTP support enabled (vLLM ≥ 0.25 or

SGLang); our verification above ran the target model only.

Provenance (revisions)

| Role | Artifact | Exact revision / hash |

| --- | --- | --- |

| Upstream official | deepseek-ai/DeepSeek-V4-Flash-0731 | 9e165c30e2704aec5d9d593cce3eebd58bbef1cb |

| Abliterated source (this bake input) | apetersson/DeepSeek-V4-Flash-0731-Abliterated-FP8 | HF commit 7d02640c72a2c8127f116d3d1933ddfec5e4c0fa (local download metadata) |

| Abliteration tooling | apetersson/deepseek-model-tools | c3979d5f8179f2a8714ac459b0f2e242006d6ad0 (scripts/abliterate_dsv4.py) |

| Refusal direction | drowzeys/DeepSeek-V4-Flash-DSpark-Abliterated-Uncensored-1M-57toks | 85ae5bbaa7daf4da7b40e007a71a1c06b41fc053 |

| Direction SHA-256 | results/refusal_direction_r1.pt | 6e4d8a8f3aa9e21795faab2c5b14d29b019acdf2ddbfbd8238430458a5837fe0 |

| Model-Optimizer package pin | nvidia-modelopt | 0.44.0 (matches NVIDIA NVFP4 card) |

| Public recipe tree | NVIDIA/Model-Optimizer tag 0.45.0 | git ec87a82927d003986d44fb7f4fa8b3d10c31b095 |

| Recipe entrypoint | examples/deepseek/deepseek_v4/quantize_to_nvfp4.py | SHA-256 f14ba206809875241b758202c565d0387338b9e1511108959da15711ef4a8feb |

| Kiln image | lna-lab/abliterated-nvfp4-dspark-kiln:v0.44.0 | sha256:b186cefbbce105ccbfd1638595292df4aa2c6159bf3d8be56bf3385249770c75 |

See also ABLITERATION_MANIFEST.json (inherited from the FP8 abliterated

source) and the staged SHA256SUMS for every file in this tree.

Quantization recipe (ignore list 1:1)

Producer stamp written into config.json / hf_quant_config.json:

{ "name": "modelopt", "version": "dsv4-nvfp4-experts" }

Hardcoded exclude / HF ignore list (public recipe, not modified):

*.attn.*
*.ffn.shared_experts.*
head
mtp.*

Meaning:

Converted: routed MoE experts only → NVFP4 (moe_quant_algo: NVFP4,

group size 16).

Not converted (passthrough): attention, shared experts, LM head, and all

mtp.* (DSpark / multi-token prediction) tensors, remaining in the source

mixed formats.

What was actually run (MEASURED bake path)

Command shape (see bake log; device was CUDA for the cast export on this host):

python3 quantize_to_nvfp4.py \
  --amax_path /path/to/amax-nvfp4-experts-abl \
  --source_ckpt /path/to/0731-abliterated-fp8 \
  --output_ckpt /path/to/0731-abliterated-nvfp4 \
  --device cuda \
  --cast_mxfp4_to_nvfp4 \
  --overwrite

Activation PTQ (ptq.py / multi-GPU torchrun+NCCL calibration) was NOT run.

On the 10×16 GB bake host the full PTQ load path does not fit; input_scale

values were stamped from a synthetic constant amax:

INPUT_AMAX = 6.0 for every routed expert projection

→ input_scale = 1/448

Weight path under --cast_mxfp4_to_nvfp4 uses the lossless MXFP4→NVFP4 cast

(weight amax not used for the casted weights).

Synthetic amax dump: 66 048 keys (33 024 input + 33 024 weight),

SHA-256 bcfcd9e53ef3bcf714b0f943448b8f6163c88f9c380297875985293089ecc6b2.

Cast statistics (MEASURED from export log)

| Metric | Value |

| --- | ---: |

| Lossless MXFP4→NVFP4 blocks | 8 657 043 456 / 8 657 043 456 (100.0000%) |

| Routed expert projections converted | 33 024 |

| Passthrough tensors | 6 269 |

| Quantized routed-expert modules (layers) | 43 |

| Weight shards | 48 |

| Indexed tensors | 138 365 |

| On-disk shards | 163.49 GiB (~175.6 GB) |

Abliteration (inherited; not re-done here)

The NVFP4 cast does not re-apply abliteration. Surgery remains exactly as in

the FP8 abliterated source:

Method: refusal-direction projection from attention residual writers.
Rank 1, λ = 3.5, layers 10–42 inclusive, plus corresponding MTP attention

wo_b writers (36 tensors total).

Three fixed-point FP8 requantization iterations at abliteration time.
Direction SHA-256:

6e4d8a8f3aa9e21795faab2c5b14d29b019acdf2ddbfbd8238430458a5837fe0.

Because attention and MTP are on the ignore / passthrough list, those edited

tensors are copied, not requantized, by this bake.

Format and use

Architecture, tokenizer, official message encoding, 1 M context declaration,

and attached DSpark tensors come from the 0731 line. Use the included

encoding/ directory and the upstream model card for prompt formatting.

Recommended upstream sampling defaults: temperature=1.0, top_p=0.95 for

agentic scenarios, top_p=1.0 otherwise.

Serving (NOT verified for this artifact on the bake host)

NVIDIA’s NVFP4 Flash card documents:

SGLang, or
vLLM ≥ 0.22 (and ≥ 0.25 for DSpark features per project notes).

The bake host’s stock vLLM 0.21.0 is not the verified stack for this

checkpoint. Do not treat local 0.21 install as a green light.

Reference NVIDIA deploy notes (for the official NVFP4 sibling, not this

abliterated tree): SGLang with TP; vLLM with --kv-cache-dtype fp8 on

Blackwell-class hardware.

Validation status

Structural validation (MEASURED on this tree)

Live re-check of the n3 structural gates against the written files:

| Gate | Result |

| --- | --- |

| ignore list 1:1 vs NVIDIA ref | PASS |

| producer dsv4-nvfp4-experts | PASS |

| moe_quant_algo=NVFP4, group_size 16 | PASS |

| quant_algo=MIXED_PRECISION | PASS |

| activation_scheme=dynamic | PASS |

| config_groups weights float4 (num_bits=4, type=float, gs=16) | PASS |

| routed weight/scale/input_scale counts 33 024 each | PASS |

| no leftover routed MXFP4 .scale keys | PASS |

| MTP present and not NVFP4-converted | PASS |

| size sane (150–175 GiB shards) | PASS |

Pilot (n3a) also showed CPU vs GPU cast byte-identical on a miniature

source and full attention/MTP passthrough integrity.

Behavioral validation of this NVFP4 artifact

NOT MEASURED. No refusal gauntlet, no StrongREJECT judge, no capability

suite, and no production serve was run against

/mnt/stripe/models/0731-abliterated-nvfp4.

Inherited / proxy evidence (about the source abliteration line only)

The FP8 abliterated source card documents an MLX mixed 2/3-bit deployment proxy

benchmark (oMLX, 2026-08-01). That evidence:

applies to a quantized MLX derivative of the abliterated line,
is not a direct measurement of native FP8,
is not a measurement of this NVFP4 tree,
must not be copied as scores for this release without re-running.

MEASURED vs ESTIMATE (explicit)

MEASURED

Source abliterated tree downloaded at HF commit

7d02640c72a2c8127f116d3d1933ddfec5e4c0fa.

Bake used public quantize_to_nvfp4.py with --cast_mxfp4_to_nvfp4 and

producer dsv4-nvfp4-experts.

Ignore list is exactly

.attn., .ffn.shared_experts., head, mtp.*.

MXFP4→NVFP4 weight cast lossless block rate 100%.
Structural gates above all PASS; disk shard total 163.49 GiB.
Attention / shared / head / MTP tensors are not NVFP4-rewritten

(abliteration residual writers preserved by passthrough).

Full per-file SHA-256 inventory of this tree (see SHA256SUMS).

ESTIMATE / synthetic / not measured

Activation scales: synthetic INPUT_AMAX=6.0 → input_scale=1/448.

Not calibrated on real traffic. Quality impact of synthetic input scales

is unknown.

Serving stack fit and tokens/s: not measured on this artifact.

Plan-level “44 t/s with MTP on 12×16 GB” remains an ESTIMATE / goal, not

a result.

Refusal / uncensor behavior after NVFP4 cast: not measured.
Capability preservation (coding, reasoning, long context): not measured.
Parity with nvidia/DeepSeek-V4-Flash-NVFP4 accuracy tables: not claimed;

NVIDIA calibrated activations; this bake did not.

Limitations

Abliteration can affect capabilities beyond refusals.
Synthetic activation scales may interact poorly with some runtimes or

sequences even when weights cast losslessly.

DSpark/MTP behavior under NVFP4+passthrough has not been load-tested here.
Use access controls appropriate to an abliterated model.

License and attribution

Upstream DeepSeek weights: MIT (LICENSE).
Refusal direction: drowzeys/keys MIT notice in NOTICE.
Quantization recipe: NVIDIA Model-Optimizer (public examples); this card is

not an NVIDIA release.

Please credit DeepSeek-AI, the abliteration source maintainers, and NVIDIA

Model-Optimizer when redistributing or publishing results.

Integrity

Publish only after Ken’s explicit approval. Irreversible Hub upload was not

performed by the n4 staging job. Verify against SHA256SUMS after any copy.

sakamakismile/DeepSeek-V4-Flash-0731-Abliterated-NVFP4

DeepSeek-V4-Flash-0731-Abliterated-NVFP4

Verified on the bake host (MEASURED)

Serving notes

DSpark (MTP) — it is inside this checkpoint

Serving DSpark today: MEASURED status (2026-08-02)

Provenance (revisions)

Quantization recipe (ignore list 1:1)

What was actually run (MEASURED bake path)

Cast statistics (MEASURED from export log)

Abliteration (inherited; not re-done here)

Format and use

Serving (NOT verified for this artifact on the bake host)

Validation status

Structural validation (MEASURED on this tree)

Behavioral validation of this NVFP4 artifact

Inherited / proxy evidence (about the source abliteration line only)

MEASURED vs ESTIMATE (explicit)

MEASURED

ESTIMATE / synthetic / not measured

Limitations

License and attribution

Integrity