DeepSeek V4 Flash 0731 IK GGUF

Mixed-precision GGUF quantizations of DeepSeek-V4-Flash-0731, made with ik_llama.cpp and an importance matrix. The ladder ranges from a sub-64-GiB build to a near-native 4-bit build.

The routed MXFP4 tensors were requantized directly from their native representation. FP8-sourced tensors were expanded to BF16 before quantization.

September 2026 file correction

XS_IQ1_KT, IQ1_KT, IQ2_KL, IQ3_KS, IQ3_K, and IQ4_KSS have been rebuilt. A Python metadata rewrite had truncated tensors with per-row headers in the previous files. Those files must be replaced. IQ2_KS and IQ2_KT were unaffected and are unchanged.

The rebuilt files embed the template during native quantization. Their tensor-region sizes and template were checked, and sampled first/last rows were dequantized and checked for finite values before transfer. Selected expert groups previously using IQ4_KS or IQ4_K now retain native MXFP4. This change does not apply to higher-precision non-expert tensors.

Compatibility warning

These files are intended for a recent build of ik_llama.cpp. They contain IK-specific IQK and IQT tensor types. Mainline llama.cpp and applications built against it, including many releases of LM Studio, KoboldCpp, and text-generation-webui, may not load them. Check that your backend explicitly supports the tensor types in the chosen file.

DeepSeek V4 Flash advertises a very large context window. Set -c explicitly unless you want the runtime to reserve a correspondingly large KV cache. IQ2_KT is slower than IQ2_KS on CPU and CPU/GPU hybrid inference despite its similar size.

Available files

Sizes below are the final release-file sizes. Leave additional memory for the KV cache, runtime buffers, the operating system, and any layers that do not fit cleanly on a GPU.

Average bpw is file-size-derived relative to the 4.555-bpw BF16/MXFP4 source. It covers the full GGUF, including non-expert tensors and format overhead.

Pick the largest file that leaves enough headroom for your intended context and offload split. File size alone is not a complete memory estimate.

Quality measurements

The chart compares the locally evaluated IK ladder with the two valid Atomic reference runs. Every plotted quant was tested against the same lossless BF16 reference distributions.

Lower PPL, KLD, and RMS delta-p are better. Higher top-1 agreement is better.

At almost the same size as Atomic AD-IQ2_S, this release's IQ2_KS has 8.35% lower mean KLD, 0.100488 lower PPL, 1.162 percentage points lower RMS delta-p, and 0.920 percentage points higher top-1 agreement. The measured IQ1_KT also has 19.03% lower mean KLD than Atomic AD-IQ1_M_XL at a similar size.

XS_IQ1_KT, IQ1_KT, and IQ2_KL were retested on the rebuilt release artifacts. The unchanged IQ2_KT and IQ2_KS retain their previous release-artifact measurements.

Test method

The KLD evaluation follows AtomicChat's published setup and uses its DeepSeek V4 evaluation artifacts:

• lossless AD-BF16 logits from Atomic's ref5632.kld as the reference distribution;
• the exact corresponding WikiText-2 tokens at context 5632;
• the first 31 of Atomic's 51 evaluation chunks;
• the same batching configuration and tensor split, with batch and microbatch sizes of 512 and seed 1337;
• perplexity, full-distribution KL divergence, RMS probability change, and reference top-1 agreement.

The rebuilt XS_IQ1_KT, IQ1_KT, and IQ2_KL were tested with IK build 4882, commit 3e416d7f5. The retained IQ2_KS, IQ2_KT, and two Atomic reference runs used build 4838, commit 7cd62a3eb. The reference tokens, BF16 logits, chunk selection, hardware, and requested test settings are unchanged, but this is not a single-executable comparison. Small differences may include runtime changes.

The two Atomic GGUFs in the table were rerun locally through the same 31-chunk harness. Do not compare these absolute values directly with Atomic's published 51-chunk table.

IQ3_KS, IQ3_K, and IQ4_KSS were not KLD-tested. They exceed the available local hardware limit for this evaluation, so no quality claim is made for them here.

Running the model

Use a current ik_llama.cpp build. For example:

./llama-server \
  -m DeepSeek-V4-Flash-0731-IQ2_KS.gguf \
  -c 8192 \
  -ngl 99 \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort":"max"}' \
  --reasoning-format deepseek

Adjust context, GPU layers, tensor split, and batch settings for your hardware. The GGUF metadata embeds the corrected Jinja template from Unsloth's safetensors repository, including its reasoning-effort handling.

Credits and provenance

• Base model: deepseek-ai/DeepSeek-V4-Flash-0731
• Quantization backend: ikawrakow/ik_llama.cpp
• Importance matrix and reference methodology: AtomicChat/DeepSeek-V4-Flash-0731-GGUF
• BF16 KLD reference dump and evaluation data: AtomicChat/dsv4-eval-artifacts
• Embedded Jinja chat template: unsloth/DeepSeek-V4-Flash-0731

Thanks to AtomicChat for publishing the imatrix, reference logits, comparison quants, and enough evaluation detail to reproduce a paired local test.

License

This release is available under the MIT License. The base model is also MIT-licensed; review the upstream model card before redistribution or deployment.