dspark-DeepSeek-V4-Pro-0813.gguf
GGUF conversion of the dspark speculative-decoding drafter from
deepseek-ai/DeepSeek-V4-Pro-0813
for use as a draft model with llama.cpp
or ik_llama.cpp's --spec-type dspark.
Files
dspark-DeepSeek-V4-Pro-0813-IQ3_XXS-imx.gguf— IQ3_XXS draft model (29 GB)
Usage
Attach to a DeepSeek-V4-Pro-0813 verifier with --spec-type dspark:n_max=5
(the dspark config advertises dspark_block_size: 5):
llama-server \
--model /path/to/DeepSeek-V4-Pro-0813-IQ3_XXS-00001-of-00015.gguf \
--model-draft /path/to/dspark-DeepSeek-V4-Pro-0813-IQ3_XXS-imx.gguf \
--spec-type dspark:n_max=5
Compatibility
The drafter is a standard DFLASH-arch GGUF loaded via --spec-type dspark, so
it should work with both llama.cpp and
ik_llama.cpp (a llama.cpp fork); the
dspark loader expects this drafter's specific tensor names (fc /
enc.output_norm / markov_w1 / markov_w2 / conf_proj).
Tested only on ik_llama.cpp, chosen specifically to fit the full 1M context
via --swa-compress (which upstream llama.cpp does not have). It has not been
tested on other inference engines.
Notes
- Only the 9 MoE expert tensors (
blk.{0,1,2}.ffn_{gate,down,up}_exps.weight, ~96% of the drafter) are IQ3_XXS (3.06 bpw); the attention / markov / fc / shexp tensors stay at their original Q8_0 / BF16 / F32. This was a deliberate pre-emptive size cut so the 29 GB drafter + ~53 GB IQ3_XXS main model fits within 96 GB VRAM with headroom (the full 41 GB MXFP4 drafter would land at ~94 GB, right at the ceiling). - The
-imxsuffix means "imatrix-quantized". IQ3_XXS requires an importance matrix; the imatrix used is the 6block main-model imatrix (6block/DeepSeek-V4-Pro-0813-GGUF→imatrix.gguf), whoseblk.0/1/2expert entries are shape-compatible with the drafter's 3 MTP layers and have clean statistics. The drafter's own target layers (main modelblk.58-60) had pathological imatrix entries (zero-count NaNs, extreme outliers) and were deliberately not used. - The tokenizer is copied verbatim from the DeepSeek-V4-Pro-0813 main-model GGUF.
Recipe
How this drafter GGUF was produced (for reproducibility / re-quantization):
Download the 3 dspark safetensors shards (all
mtp.*-prefixed tensors) + config/tokenizer fromdeepseek-ai/DeepSeek-V4-Pro-0813:model-00064-of-00066.safetensors,model-00065-of-00066.safetensors,model-00066-of-00066.safetensorsWrite a trimmed
model.safetensors.index.jsoncontaining only the 7009mtp.*tensors so the converter opens only the 3 shards.Convert to BF16 GGUF with upstream llama.cpp's
convert_hf_to_gguf.py(--dsparkis only in upstream ggml-org/llama.cpp, not ik_llama.cpp):python convert_hf_to_gguf.py <dir> --outtype bf16 \ --outfile dspark-DeepSeek-V4-Pro-0813-BF16.gguf \ --dspark --target-model-dir <dir> --model-name DeepSeek-V4-Pro-0813→ 81 tensors, 42.1 GB BF16.
Quantize with ik_llama.cpp
llama-quantize, requantizing only the 9 expert tensors to IQ3_XXS (imatrix required) and pinning every other weight to its current type via a--tensor-type-file:# expert-tt.txt: 9 experts -> IQ3_XXS; all other weights -> current type llama-quantize --allow-requantize \ --imatrix imatrix-6block-main.gguf \ --tensor-type-file expert-tt.txt \ dspark-DeepSeek-V4-Pro-0813-BF16.gguf \ dspark-DeepSeek-V4-Pro-0813-IQ3_XXS-imx.gguf IQ3_XXS 64→ 29 GB (9 experts IQ3_XXS @ 3.24 GB each; ~1.6 GB rest preserved).
Hardware
Tested on:
- GPU: 1× NVIDIA RTX Pro 6000 (96 GB VRAM)
- CPU: 2× AMD EPYC 9355 (64 cores / 128 threads)
- RAM: 768 GB DDR5
Deployment
Adjust the ${modelsPath} prefix to wherever you download the weights — place
both the base-model shards and the drafter under
${modelsPath}/DeepSeek-V4-Pro-0813/:
/etc/ik-llama-cpp/llama-server
--model ${modelsPath}/DeepSeek-V4-Pro-0813/DeepSeek-V4-Pro-0813-IQ3_XXS-00001-of-00015.gguf
--model-draft ${modelsPath}/DeepSeek-V4-Pro-0813/dspark-DeepSeek-V4-Pro-0813-IQ3_XXS-imx.gguf
--spec-type dspark:n_max=5
--jinja
-amb 512
-ctk f16
-ctv f16
-ngl 99
--cpu-moe
--swa-compress
--ctx-size 1000000
--batch-size 8192
--ubatch-size 8192
--threads 64
--ctx-checkpoints 12
Performance
Measured with ik_llama.cpp (IQ3_XXS main model, 1M ctx):
- Draft acceptance ~46% aggregate (range ~27-80% per request).
- Generation speed up to ~19 t/s vs ~10 t/s without the drafter (~1.3-1.9x).