Qwen3.8-Flash-Next-DFlash2-GGUF

GGUF conversions of incoai/Qwen3.8-Flash-Next-DFlash2, the DFlash 2 draft model for Qwen/Qwen3.8-Flash-Next, for use with llama.cpp's draft-dflash speculative decoding.

Decode speed on a UD-Q4_K_XL target, -c 262144, Apple M5 Max (Metal). Method under Measured performance.

This is not a standalone language model. It drafts tokens that the Qwen3.8-Flash-Next target verifies. DFlash 2 is a block-diffusion drafter: it proposes a whole block of tokens in one pass, keeps the top candidates at each position, and a small selector picks one coherent path through them. Verification is lossless: greedy output matches the target exactly and sampling preserves the target's distribution. See the upstream card, the blog post and z-lab/dflash for the method and the upstream (SGLang, GB300) evaluation.

Files

Drafter: 5 layers, hidden 2560, block size 8, sliding-window attention (2048), reads target layers 5/15/25/35/45. The BF16 source is 1,150 MiB.

Recommendation: use Q4_0

Q4_0 matches Q8_0 on acceptance and speed and loads 287 MiB less. Measured with the same target and settings (n-max 5, p-min 0.75, 2 passes):

Memory: MTP head vs DFlash 2

On disk the DFlash 2 drafter is about 6x smaller than Flash-Next's MTP head. At runtime the two cost about the same, because the DFlash drafter's compute buffer (2.9 GiB) is much larger than its weights. Measured from llama-server's own buffer reports: Qwen3.8-Flash-Next UD-Q5_K_XL target, -c 131072, -ub 1024, Metal.

Two scaling notes:

• The DFlash KV cache is fixed at 60 MiB because its attention is a 2048-token sliding window. The MTP head's KV cache grows with -c; by linear extrapolation it is about 640 MiB at 262144 (not measured).
• Flash-Next is a hybrid (Gated DeltaNet) model. The target keeps one recurrent-state snapshot per draft token so it can roll back rejected drafts, so a deeper --spec-draft-n-max costs about 113 MiB per step whichever drafter you use.

llama.cpp usage

Recommended settings: draft-dflash, n-max 5, p-min 0.75, default (greedy) draft sampling, one slot.

llama-server \
  -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
  -md Qwen3.8-Flash-Next-DFlash2-Q4_0.gguf \
  --spec-type draft-dflash --spec-draft-n-max 5 --spec-draft-p-min 0.75 \
  -c 131072 -np 1 -ub 1024 -ngl 99 -fa on \
  --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0

Add --mmproj <mmproj file> for vision; DFlash keeps drafting on image prompts. The sampling values are Flash-Next's official thinking profile.

Notes:

• Requires llama.cpp v0.8.0 or newer for any DFlash drafter on a Qwen3.8-Flash-Next target. Earlier builds abort on the first request; #30927 fixed it.
• n-max and p-min are fixed at launch. The draft-dflash path ignores per-request speculative.n_max / speculative.p_min, so changing them means restarting the server.
• Depth barely matters at p-min 0.75. n-max 5 / 6 / 7 / 8 measured 74.5 / 75.2 / 75.8 / 76.2 t/s (geomean), which is a tie. Deeper drafts only help list-like output (+29% at n5, +48% at n8 vs MTP) and slightly hurt prose (-1% at n5, -5% at n8). Use n-max 8 only if your workload is list-heavy.
• Gating: p-min 0.75 beat 0.3 (72.6) and 0.0 (69.6). p-min 0.9 loses (70.9). --spec-draft-sampling probabilistic measured -0.7% against greedy, so keep the default.
• Cascading with ngram-mod (--spec-type ngram-mod,draft-dflash) can produce drafts longer than n-max. On hybrid targets that needs llama.cpp b11996 or newer, which includes #31350. Plain draft-dflash drafts never exceed n-max.
• Concurrency: everything here was measured at -np 1. Multi-slot behaviour on Metal was not tested.

Measured performance

Hardware and build: Apple M5 Max, 128 GB, Metal. llama.cpp v0.8.0 (8642541b2) + #31350 + #29030 (row reads for the n-gram table; memory only, unrelated to DFlash).

Method: temperature 1.0, top-p 0.95, top-k 20, thinking off, 600 new tokens, fixed seeds, 2 passes. The MTP baseline was run both before and after the DFlash arms in the same session, and the two MTP runs agreed within 3%. Prompts: prose (a networking explanation), code (an LRU cache module with tests), list (Fibonacci and prime listings), and ctx (a 5.8k-token C++ source excerpt to rewrite with comments).

Qwen3.8-Flash-Next (Unsloth UD-Q5_K_XL), -c 131072

Acceptance: MTP 89.7% vs DFlash 87.4% per drafted token. DFlash drafts more tokens per verification step, which is where the speedup comes from.

Fine-tunes of the same base (drafter unchanged)

The drafter was trained on the base model, but it also works on post-trained variants. Same method, MTP run first and then DFlash:

Every target passed a tool call, a thinking-mode answer, and two image prompts with DFlash on.

Conversion

Both files were converted with llama.cpp's convert_hf_to_gguf.py. The target-tokenizer folder shipped in the upstream repo supplies the vocabulary:

python convert_hf_to_gguf.py Qwen3.8-Flash-Next-DFlash2 \
  --target-model-dir Qwen3.8-Flash-Next-DFlash2/target-tokenizer \
  --outtype bf16 --outfile Qwen3.8-Flash-Next-DFlash2-BF16.gguf
llama-quantize Qwen3.8-Flash-Next-DFlash2-BF16.gguf Qwen3.8-Flash-Next-DFlash2-Q4_0.gguf Q4_0

The Q8_0 was made with the v0.7.0 converter. Re-converting it with v0.8.0 gives identical tensors.

Credits and citation

The drafter is by Inco AI, built on DFlash. Only the GGUF conversion and the llama.cpp measurements are from this repo. Please cite the original work:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

@inproceedings{chen2026dflash,
  title     = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}