gemma4-12B-it-DFlash-GGUF

GGUF conversion of z-lab/gemma4-12B-it-DFlash for use with llama.cpp.

This is a DFlash draft model, not a standalone language model.

It must be used together with a compatible google/gemma-4-12B-it GGUF target model.

Model Details

• Source model: z-lab/gemma4-12B-it-DFlash
• Compatible target: google/gemma-4-12B-it
• Format: GGUF
• Quantization: Q4_K_M
• Purpose: DFlash speculative decoding

This repository contains only the DFlash draft model. It does not include the Gemma 4 12B target model.

Compatibility

A recent version of llama.cpp with DFlash support is required.

Tested with:

• llama.cpp b9831
• NVIDIA GeForce RTX 5070 Ti 16 GB
• Target model: gemma-4-12b-it-UD-Q4_K_XL

Other llama.cpp builds, target quantizations, hardware configurations, prompts, and sampling settings may produce different performance.

Usage

llama-server

llama-server \
  -m gemma-4-12b-it-UD-Q4_K_XL.gguf \
  -md gemma4-12B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4

llama-cli

llama-cli \
  -m gemma-4-12b-it-UD-Q4_K_XL.gguf \
  -md gemma4-12B-it-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4

The target and draft models must use the same tokenizer and vocabulary.

Suggested Settings

Based on the benchmark below:

• n_max = 4 gave the best aggregate wall-clock result.
• n_max = 2 gave the highest overall acceptance rate and performed better on several high-entropy natural-language tasks.
• n_max = 4–5 performed well for predictable outputs such as code completion, JSON, and repeated patterns.
• n_max = 5 improved a few highly structured tasks, but was slower overall than n_max = 4 in this mixed benchmark.

A reasonable general starting point is:

--spec-draft-n-max 4

For translation, creative writing, explanations, or conversational output, n_max = 2 may be preferable. For JSON and highly predictable output, n_max = 4–5 may perform better. For general code completion, n_max = 3–4 may be a better starting point.

Conversion

Converted from the original Hugging Face DFlash checkpoint using convert_hf_to_gguf.py.

Gemma 4 tokenizer metadata was loaded from the compatible target model directory through --target-model-dir.

No neural-network weights were edited or retrained. The weights were converted and quantized to GGUF format.

Benchmark

Benchmarked using mtp-bench on an NVIDIA GeForce RTX 5070 Ti.

Target model:

gemma-4-12b-it-UD-Q4_K_XL

Draft model:

gemma4-12B-it-DFlash-Q4_K_M

Aggregate Results

These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.

Performance varies substantially by task type.

Detailed Results

n_max = 2

code_python        pred= 192 draft= 157 acc= 112 rate=0.713 tok/s=112.2
code_cpp           pred= 192 draft= 152 acc= 115 rate=0.757 tok/s=123.0
explain_concept    pred= 192 draft= 190 acc=  95 rate=0.500 tok/s=98.1
summarize          pred=  43 draft=  38 acc=  24 rate=0.632 tok/s=110.2
qa_factual         pred= 160 draft= 148 acc=  85 rate=0.574 tok/s=104.5
translation        pred=  26 draft=  32 acc=  10 rate=0.312 tok/s=78.2
creative_short     pred=  44 draft=  56 acc=  15 rate=0.268 tok/s=77.0
stepwise_math      pred= 192 draft= 145 acc= 118 rate=0.814 tok/s=128.3
json_output        pred= 192 draft= 134 acc= 124 rate=0.925 tok/s=134.3
long_reasoning     pred= 192 draft= 155 acc= 113 rate=0.729 tok/s=115.2
repeat_pattern     pred= 192 draft= 130 acc= 126 rate=0.969 tok/s=141.6
code_completion    pred= 166 draft= 128 acc= 102 rate=0.797 tok/s=125.2
long_code_review   pred= 192 draft= 203 acc=  89 rate=0.438 tok/s=88.7

Aggregate:
  requests:            13
  predicted tokens:    1975
  draft tokens:        1668
  accepted tokens:     1128
  acceptance rate:     0.6763
  total wall time:     19.50 s

n_max = 3

code_python        pred= 192 draft= 206 acc= 122 rate=0.592 tok/s=114.0
code_cpp           pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=122.0
explain_concept    pred= 192 draft= 261 acc= 103 rate=0.395 tok/s=92.3
summarize          pred=  43 draft=  48 acc=  27 rate=0.562 tok/s=112.3
qa_factual         pred= 160 draft= 201 acc=  93 rate=0.463 tok/s=98.0
translation        pred=  26 draft=  45 acc=  10 rate=0.222 tok/s=69.8
creative_short     pred=  44 draft=  78 acc=  18 rate=0.231 tok/s=70.4
stepwise_math      pred= 192 draft= 186 acc= 129 rate=0.694 tok/s=130.2
json_output        pred= 192 draft= 155 acc= 139 rate=0.897 tok/s=152.4
long_reasoning     pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=122.0
repeat_pattern     pred= 192 draft= 146 acc= 142 rate=0.973 tok/s=162.2
code_completion    pred= 166 draft= 156 acc= 114 rate=0.731 tok/s=132.7
long_code_review   pred= 192 draft= 285 acc=  94 rate=0.330 tok/s=79.3

Aggregate:
  requests:            13
  predicted tokens:    1975
  draft tokens:        2159
  accepted tokens:     1241
  acceptance rate:     0.5748
  total wall time:     19.29 s

n_max = 4

code_python        pred= 192 draft= 257 acc= 126 rate=0.490 tok/s=112.7
code_cpp           pred= 192 draft= 220 acc= 135 rate=0.614 tok/s=132.7
explain_concept    pred= 192 draft= 304 acc= 114 rate=0.375 tok/s=96.4
summarize          pred=  46 draft=  68 acc=  29 rate=0.426 tok/s=102.9
qa_factual         pred= 162 draft= 248 acc= 100 rate=0.403 tok/s=101.8
translation        pred=  26 draft=  56 acc=  12 rate=0.214 tok/s=70.8
creative_short     pred=  44 draft= 100 acc=  19 rate=0.190 tok/s=68.8
stepwise_math      pred= 192 draft= 219 acc= 136 rate=0.621 tok/s=133.6
json_output        pred= 192 draft= 176 acc= 146 rate=0.830 tok/s=163.8
long_reasoning     pred= 192 draft= 228 acc= 133 rate=0.583 tok/s=127.3
repeat_pattern     pred= 192 draft= 163 acc= 149 rate=0.914 tok/s=174.7
code_completion    pred= 162 draft= 196 acc= 113 rate=0.577 tok/s=127.3
long_code_review   pred= 192 draft= 347 acc= 104 rate=0.300 tok/s=80.2

Aggregate:
  requests:            13
  predicted tokens:    1976
  draft tokens:        2582
  accepted tokens:     1316
  acceptance rate:     0.5097
  total wall time:     18.81 s

n_max = 5

code_python        pred= 192 draft= 285 acc= 133 rate=0.467 tok/s=112.0
code_cpp           pred= 192 draft= 285 acc= 134 rate=0.470 tok/s=116.8
explain_concept    pred= 192 draft= 384 acc= 113 rate=0.294 tok/s=85.5
summarize          pred=  46 draft=  85 acc=  29 rate=0.341 tok/s=92.6
qa_factual         pred= 162 draft= 310 acc= 100 rate=0.323 tok/s=90.3
translation        pred=  26 draft=  70 acc=  12 rate=0.171 tok/s=62.9
creative_short     pred=  44 draft= 125 acc=  19 rate=0.152 tok/s=58.6
stepwise_math      pred= 192 draft= 257 acc= 139 rate=0.541 tok/s=124.4
json_output        pred= 192 draft= 182 acc= 154 rate=0.846 tok/s=171.4
long_reasoning     pred= 192 draft= 274 acc= 135 rate=0.493 tok/s=115.3
repeat_pattern     pred= 192 draft= 173 acc= 155 rate=0.896 tok/s=175.2
code_completion    pred= 162 draft= 225 acc= 117 rate=0.520 tok/s=120.2
long_code_review   pred= 192 draft= 470 acc=  94 rate=0.200 tok/s=67.8

Aggregate:
  requests:            13
  predicted tokens:    1976
  draft tokens:        3125
  accepted tokens:     1334
  acceptance rate:     0.4269
  total wall time:     20.45 s

Observations

• n_max = 2 achieved the highest aggregate draft acceptance rate.
• n_max = 4 achieved the shortest total wall time in this mixed benchmark.
• n_max = 3 was close to n_max = 4 while using fewer draft tokens.
• n_max = 5 improved JSON output and repeated-pattern generation, but increased total wall time.
• Translation, creative writing, conceptual explanations, and long-form code review showed substantially lower acceptance as n_max increased.
• General code completion peaked at n_max = 3 in this benchmark, while JSON and repeated patterns benefited from longer draft sequences.

Notes

This repository contains only the DFlash draft model.

A compatible google/gemma-4-12B-it GGUF target model is required. The target GGUF may use a different quantization from the draft model, but both models must share compatible tokenizer and model architecture assumptions.

The benchmark results are specific to the tested hardware, model quantizations, llama.cpp build, prompts, and runtime settings.

Credits

• Z Lab — DFlash method and original draft checkpoint
• Google DeepMind — Gemma 4 and google/gemma-4-12B-it
• ggml-org — llama.cpp, GGUF, and DFlash inference support

License

This repository contains a converted and quantized GGUF version of the original DFlash draft checkpoint.

The upstream z-lab/gemma4-12B-it-DFlash and google/gemma-4-12B-it repositories identify their applicable licensing terms. Users should review the upstream model cards and licenses before redistribution or commercial use.