A family of seven GGUF files of DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic (DeepSeek-V4-Pro distill of Qwen3.5-9B: 32-block hybrid, 24 linear-attention + 8 full-attention blocks), converted from esatapedico/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4. The MTP (multi-token prediction) speculative head is baked into every file — no separate drafter needed.
VERY-LOW / COMPACT-LOW / LOW / MEDIUM / MID-HIGH / HIGH / VERY-HIGH — a compact family sharing a byte-identical 224-tensor NVFP4 backbone (96 MLP + 32 full-attention q/k/v/o packed NVFP4 from the checkpoint, plus 96 re-quantized linear-attention gate/qkv/beta/out tensors), differing only in the head tensors (LM head, token embedding, MTP draft head). COMPACT-LOW fills the gap between VERY-LOW and LOW. MID-HIGH sits between MEDIUM and HIGH with all three head groups at Q8_0.The goal: keep native NVFP4 density across the whole model for Blackwell, and offer a size/precision ladder for the tensors that most affect output quality and decode speed.
These GGUFs are text-only on their own. Vision is enabled by pairing any tier with the mmproj-...-f16.gguf projector shipped in this repo (converted from the BF16 source vision tower with convert_hf_to_gguf.py --mmproj, 334 tensors: 112 F16 + 222 F32) via --mmproj.
I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.
🌐 esatapedico.com. Updates, work-in-progress, and ways to support.
| File | Size | SHA-256 | lm_head (output.weight) |
token_embd | MTP head (blk.32) | Attention |
|---|---|---|---|---|---|---|
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-LOW.gguf |
4,768,390,336 bytes | eb3ec611a53cc08f06cc058276730699d8cb00ee0bcc0cb6e3b31ced193a7ae4 |
Q3_K |
Q2_K |
Q2_K |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-COMPACT-LOW.gguf |
5,006,777,536 bytes | a53a225a0d58604328b44151fd36b13b32167b5d86d0986bb6aa1e3da9c107e4 |
Q4_K |
Q3_K |
Q2_K |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-LOW.gguf |
5,286,632,608 bytes | fbfe6ff76af42e3a932d3fd1aea644eb739eae4f4256c41aa05ce2cffd3da4f3 |
Q5_0 |
IQ4_XS |
IQ4_XS |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MEDIUM.gguf |
5,962,063,040 bytes | 4d891dfc7fc4f2d5955b19d8ee5978781e8d6826a6f2fded4abda45774a22f68 |
Q8_0 |
Q6_K |
IQ4_XS |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MID-HIGH.gguf |
6,337,633,472 bytes | 139776d7aae7da029f38fec3bf6d9ca5d952ddc6e15e8978c83e6cefef43d353 |
Q8_0 |
Q8_0 |
Q8_0 |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-HIGH.gguf |
6,915,611,808 bytes | c6ff34cc3b3907a123460cf0192988856e76712230d71eb8ea8d921709816c45 |
BF16 |
Q6_K |
IQ4_XS |
NVFP4 |
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-HIGH.gguf |
8,472,796,352 bytes | 5be3d518e442ec04d388a25cd49a0580e23b4bbaca5b83cd290c98c719a2999a |
BF16 |
BF16 |
BF16 |
NVFP4 |
mmproj-DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-f16.gguf |
918,165,440 bytes | 0a0d985dfb7636a7bf556b18873f05f771e4f838152fe8a4c66d44dde407ebea |
— (vision projector) |
570 tensors per file: 312 F32 (norms, biases, scales companions) + 130 BF16 (linear-attention SSM path, embeddings where pinned) + 128 checkpoint-native NVFP4 passed through byte-identical from ORIG + 96 re-quantized at tier build (24 blocks × attn_gate, attn_qkv, ssm_beta, ssm_out). The 24 tiny per-layer ssm_alpha vectors stay F32 in every file. Cross-tier backbone identity verified: all 224 NVFP4 tensors pairwise byte-identical across all seven files (SHA-256 per tensor).
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-LOW.gguf (4,768,390,336 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-COMPACT-LOW.gguf (5,006,777,536 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-LOW.gguf (5,286,632,608 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MEDIUM.gguf (5,962,063,040 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MID-HIGH.gguf (6,337,633,472 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-HIGH.gguf (6,915,611,808 bytes)DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-HIGH.gguf (8,472,796,352 bytes)overrides-very-low.txt, overrides-compact-low.txt, overrides-low.txt, overrides-medium.txt, overrides-mid-high.txt, overrides-high.txt, overrides-very-high.txt — per-tensor quantization maps (570 entries each) for reproductionmmproj-DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-f16.gguf (918,165,440 bytes) — vision projector, converted from the BF16 source towereb3ec611a53cc08f06cc058276730699d8cb00ee0bcc0cb6e3b31ced193a7ae4 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-LOW.gguf
a53a225a0d58604328b44151fd36b13b32167b5d86d0986bb6aa1e3da9c107e4 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-COMPACT-LOW.gguf
fbfe6ff76af42e3a932d3fd1aea644eb739eae4f4256c41aa05ce2cffd3da4f3 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-LOW.gguf
4d891dfc7fc4f2d5955b19d8ee5978781e8d6826a6f2fded4abda45774a22f68 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MEDIUM.gguf
139776d7aae7da029f38fec3bf6d9ca5d952ddc6e15e8978c83e6cefef43d353 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MID-HIGH.gguf
5be3d518e442ec04d388a25cd49a0580e23b4bbaca5b83cd290c98c719a2999a DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-HIGH.gguf
c6ff34cc3b3907a123460cf0192988856e76712230d71eb8ea8d921709816c45 DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-VERY-HIGH.gguf
0a0d985dfb7636a7bf556b18873f05f771e4f838152fe8a4c66d44dde407ebea mmproj-DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-f16.gguf
GSM8K-200, MMLU-3000 (limit mode, seed 42), IFEval gate subset, ToolCap, LongQA 70k-doc, PPL on diverse text, plus cross-tier KLD anchored on VERY-HIGH (8 KB shared slice). Identical flags every tier, two 16 GB Blackwell cards.
| Tier | Size | PPL | GSM8K | MMLU-3000 | IFEval | ToolCap | LongQA | KLD vs VERY-HIGH |
|---|---|---|---|---|---|---|---|---|
| VERY-LOW | 4.44 GB | 4.5263 | 0.890 | 0.6394 | 30/30 | 100% | 0/3 | 0.0865 |
| COMPACT-LOW | 4.66 GB | 4.4294 | 0.885 | 0.6625 | 30/30 | 100% | 0/3 | 0.0680 |
| LOW | 4.92 GB | 4.4324 | 0.895 | 0.6764 | 30/30 | 100% | 0/3 | 0.0657 |
| MEDIUM | 5.55 GB | 4.4237 | 0.905 | 0.6831 | 30/30 | 100% | 0/3 | 0.0610 |
| MID-HIGH | 5.90 GB | 4.4259 | 0.875 | 0.6809 | 30/30 | 100% | 1/3 | 0.0574 |
| HIGH | 6.44 GB | 4.4243 | 0.895 | 0.6801 | 30/30 | 100% | 0/3 | 0.0609 |
| VERY-HIGH | 7.89 GB | 4.4180 | 0.890 | 0.6723 | 30/30 | 100% | 0/3 | anchor |
Quality saturates at MEDIUM: GSM8K stays in a 0.875–0.905 band across the ladder and MMLU gains are mostly realized by LOW. The KLD ladder is monotonic and an order of magnitude inside the 0.15 similarity band. LongQA 70k-doc retrieval is the weak axis (0–1/3); row-level inspection shows the first question answered correctly but grader-marked wrong, and the other two hit the harness token cap — a harness/model-format question, not a tier-quantization effect (all tiers share the byte-identical backbone).
| Tier | pp512 tok/s | tg128 tok/s |
|---|---|---|
| VERY-LOW | ~6,300–6,800 | ~92–93 |
| COMPACT-LOW | 7,595 | 88.8 |
| LOW | 6,166 | 87.0 |
| MEDIUM | 5,798 | 78.9 |
| MID-HIGH | 6,090 | 80.6 |
| HIGH | 5,785 | 67.8 |
| VERY-HIGH | 6,263 | 67.5 |
Prompt-processing varies run to run (±15% observed across repeats); decode (tg128) falls with size as expected.
llama-server -m DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4-MTP-MEDIUM.gguf -c 8192 -fa on --jinja -np 1 --port 8080
MEDIUM is the quality/size sweet spot (GSM8K 0.905, MMLU 0.683 at 5.55 GB). MTP speculative decoding is available in every file with a capable backend (-md draft model flags per llama.cpp docs).
Self-conversion of the BF16 source to NVFP4 (llm-compressor, datafree W4A16, classic recipe: 128 packed tensors), GGUF conversion with native NVFP4 preserved, then per-tier re-quantization of heads and the linear-attention path with per-tensor override maps (shipped above for reproduction). Full procedure in the build log; every tier gated on tensor census, cross-tier backbone byte-identity, smoke, PPL, speed, and the unified quality suite before release.
Derivative work, Apache-2.0 throughout; nothing here was trained or fine-tuned. Credit:
Apache-2.0, matching the source.
This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (sizes, configs, commands) against the files before relying on it.
有文件?提交种子。
提交种子