Gemma-4-26B-A4B-NVFP4-GGUF

GGUF conversion of nvidia/Gemma-4-26B-A4B-NVFP4 for use with llama.cpp.

This is an NVFP4-quantized standalone Gemma 4 target model.

It can run by itself. For the speculative-decoding results below, it was paired with a compatible Gemma 4 26B A4B Assistant/MTP draft model.

Model Details

• Source model: nvidia/Gemma-4-26B-A4B-NVFP4
• Base model: google/gemma-4-26B-A4B-it
• Format: GGUF
• Quantization: NVIDIA NVFP4 / ModelOpt
• Quantization recipe: nvfp4_experts_only
• Architecture: Gemma 4 Mixture-of-Experts with hybrid attention
• Total parameters: 25.2B
• Active parameters: 3.8B per token
• Layers: 30
• Experts: 128 routed experts, 8 active per token, plus 1 shared expert
• Context length: 256K tokens
• Vocabulary: approximately 262K tokens
• Modalities in the upstream model: text and image
• Purpose: Local inference and Assistant/MTP speculative decoding with llama.cpp

The upstream NVIDIA checkpoint was quantized with NVIDIA Model Optimizer v0.43.0. NVIDIA describes it as using an nvfp4_experts_only recipe, so not every tensor in the checkpoint is expected to use NVFP4.

This repository contains the complete target language model. The Assistant/MTP draft model used for speculative decoding is separate.

Compatibility

A recent version of llama.cpp with Gemma 4 MoE, NVFP4, and Gemma 4 Assistant/MTP support is required.

Tested with:

• Windows
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp CUDA backend
• A compatible Gemma 4 26B A4B Assistant GGUF draft model
• --spec-type draft-mtp

Older llama.cpp builds may fail to recognize the nvfp4 tensor type, load the Gemma 4 Assistant architecture, or initialize the shared KV-cache layout correctly.

Performance varies with the llama.cpp build, target and draft quantizations, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.

Multimodal inference requires a compatible projector and runtime support. The text-only benchmark below does not test image input.

Usage

Replace the draft filename below with your compatible Gemma 4 26B A4B Assistant GGUF.

llama-server

llama-server \
  -m Gemma-4-26B-A4B-NVFP4.gguf \
  -md gemma-4-26B-A4B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --flash-attn on \
  -ngl 99

llama-cli

llama-cli \
  -m Gemma-4-26B-A4B-NVFP4.gguf \
  -md gemma-4-26B-A4B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --flash-attn on \
  -ngl 99

Multi-GPU example

llama-server \
  -m Gemma-4-26B-A4B-NVFP4.gguf \
  -md gemma-4-26B-A4B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --split-mode layer \
  --tensor-split 1,1 \
  --flash-attn on \
  -ngl 99

The target and Assistant models must use compatible tokenizer, vocabulary, and Gemma 4 architecture assumptions.

An even tensor split is only a starting point. The best split depends on GPU speed, available VRAM, PCIe topology, context length, and placement of the target model, draft model, and KV cache.

Suggested Settings

Based on the mixed-task Assistant benchmark below:

• n_max = 4 gave the best aggregate throughput and shortest wall time.
• n_max = 2 gave the highest aggregate acceptance rate and remained preferable for several high-entropy tasks.
• n_max = 5 was nearly tied with n_max = 4 overall and achieved the highest speeds for repeated patterns, JSON, code completion, and stepwise mathematics.
• Increasing from n_max = 4 to n_max = 5 added substantially more draft work but did not improve aggregate throughput.
• Open-ended explanations, creative writing, and long code review generally favored shorter drafts.

Recommended general-purpose setting:

--spec-draft-n-max 4

For role-play, creative writing, open-ended explanations, conversational output, or long code review:

--spec-draft-n-max 2

For JSON, repeated patterns, deterministic code completion, and other highly predictable output:

--spec-draft-n-max 5

Conversion

Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:

python convert_hf_to_gguf.py \
  /path/to/Gemma-4-26B-A4B-NVFP4 \
  --outfile Gemma-4-26B-A4B-NVFP4.gguf \
  --outtype auto \
  --verbose

Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.

The compatible Assistant model must be converted separately. No target-model weights were retrained by this conversion.

File Size

The converted GGUF may be larger than the original Hugging Face ModelOpt checkpoint.

The conversion does not copy the original safetensors byte-for-byte. NVFP4 tensors are repacked into GGML-compatible layouts, and GGUF stores the associated scale metadata required by llama.cpp. Higher-precision tensors are also retained where required by the source checkpoint.

A larger GGUF does not mean the model was converted to FP16 or to a conventional Q4 format.

Windows reports file sizes using binary units even though File Explorer labels them as GB. A file shown with a larger decimal-GB value on Hugging Face can therefore appear smaller in Windows.

File size is not the same as runtime VRAM usage. Additional memory is needed for:

• the Assistant draft model
• target and draft KV caches
• CUDA context
• compute buffers
• graph workspace
• multimodal projector, when used

Verifying the GGUF

The converted target can be checked with llama-gguf:

.\llama-gguf.exe E:\HF_MODELS\Gemma-4-26B-A4B-NVFP4.gguf r

A successful conversion should show NVFP4 expert tensors together with their associated scale tensors. Other tensors may remain in BF16, FP16, FP8, or other source-compatible types because NVIDIA used an experts-only NVFP4 recipe.

Example:

tensor: name = blk.0.ffn_*.weight,       type = nvfp4
tensor: name = blk.0.ffn_*.scale,        type = f32
tensor: name = blk.0.ffn_*.input_scale,  type = f32

This confirms that NVFP4 tensors were preserved and that the output is not an empty metadata-only GGUF.

Benchmark

Benchmarked using mtp-bench with a compatible Gemma 4 Assistant model and:

--spec-type draft-mtp

Aggregate Results — Assistant/MTP

Effective batch throughput is calculated as total predicted tokens divided by total wall time.

Per-request tok/s can be higher because aggregate wall time may include prompt processing, request transitions, and benchmark overhead.

The predicted-token totals differ slightly because some generated responses terminated at different lengths. The n_max = 4 and n_max = 5 runs both generated 2,116 tokens and can be compared directly.

These tests do not include a non-speculative baseline and therefore should not be interpreted as direct speedup ratios over standard decoding.

Detailed Assistant Results

n_max = 2

code_python        pred= 192 draft= 139 acc= 121 rate=0.871 tok/s=144.7
code_cpp           pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=138.9
explain_concept    pred= 192 draft= 158 acc= 111 rate=0.703 tok/s=128.2
summarize          pred=  52 draft=  44 acc=  30 rate=0.682 tok/s=125.8
qa_factual         pred= 192 draft= 166 acc= 108 rate=0.651 tok/s=125.1
translation        pred= 129 draft= 102 acc=  77 rate=0.755 tok/s=135.5
creative_short     pred=  34 draft=  32 acc=  17 rate=0.531 tok/s=114.0
stepwise_math      pred= 192 draft= 137 acc= 122 rate=0.890 tok/s=145.5
json_output        pred= 192 draft= 128 acc= 127 rate=0.992 tok/s=158.4
long_reasoning     pred= 192 draft= 142 acc= 119 rate=0.838 tok/s=140.9
repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=159.1
code_completion    pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=152.5
long_code_review   pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=118.0

Aggregate:
  requests:            13
  predicted tokens:    2135
  draft tokens:        1622
  accepted tokens:     1307
  acceptance rate:     0.8058
  total wall time:     16.77 s

n_max = 3

code_python        pred= 192 draft= 162 acc= 137 rate=0.846 tok/s=167.6
code_cpp           pred= 192 draft= 179 acc= 131 rate=0.732 tok/s=150.7
explain_concept    pred= 192 draft= 209 acc= 120 rate=0.574 tok/s=127.7
summarize          pred=  52 draft=  54 acc=  33 rate=0.611 tok/s=128.2
qa_factual         pred= 192 draft= 207 acc= 121 rate=0.585 tok/s=130.1
translation        pred= 123 draft= 123 acc=  82 rate=0.667 tok/s=141.6
creative_short     pred=  34 draft=  39 acc=  21 rate=0.538 tok/s=122.6
stepwise_math      pred= 192 draft= 159 acc= 137 rate=0.862 tok/s=166.7
json_output        pred= 192 draft= 146 acc= 142 rate=0.973 tok/s=181.3
long_reasoning     pred= 192 draft= 188 acc= 128 rate=0.681 tok/s=143.5
repeat_pattern     pred= 192 draft= 143 acc= 143 rate=1.000 tok/s=188.4
code_completion    pred= 192 draft= 151 acc= 140 rate=0.927 tok/s=175.9
long_code_review   pred= 192 draft= 209 acc= 121 rate=0.579 tok/s=125.9

Aggregate:
  requests:            13
  predicted tokens:    2129
  draft tokens:        1969
  accepted tokens:     1456
  acceptance rate:     0.7395
  total wall time:     15.58 s

n_max = 4

code_python        pred= 192 draft= 176 acc= 146 rate=0.830 tok/s=179.8
code_cpp           pred= 192 draft= 201 acc= 140 rate=0.697 tok/s=162.6
explain_concept    pred= 192 draft= 277 acc= 120 rate=0.433 tok/s=116.2
summarize          pred=  52 draft=  60 acc=  38 rate=0.633 tok/s=146.9
qa_factual         pred= 192 draft= 250 acc= 128 rate=0.512 tok/s=132.7
translation        pred= 110 draft= 120 acc=  82 rate=0.683 tok/s=158.2
creative_short     pred=  34 draft=  48 acc=  23 rate=0.479 tok/s=121.9
stepwise_math      pred= 192 draft= 176 acc= 147 rate=0.835 tok/s=187.2
json_output        pred= 192 draft= 160 acc= 151 rate=0.944 tok/s=203.1
long_reasoning     pred= 192 draft= 204 acc= 140 rate=0.686 tok/s=159.4
repeat_pattern     pred= 192 draft= 153 acc= 152 rate=0.994 tok/s=211.5
code_completion    pred= 192 draft= 164 acc= 150 rate=0.915 tok/s=199.2
long_code_review   pred= 192 draft= 272 acc= 123 rate=0.452 tok/s=119.7

Aggregate:
  requests:            13
  predicted tokens:    2116
  draft tokens:        2261
  accepted tokens:     1540
  acceptance rate:     0.6811
  total wall time:     14.76 s

n_max = 5

code_python        pred= 192 draft= 205 acc= 150 rate=0.732 tok/s=182.1
code_cpp           pred= 192 draft= 240 acc= 143 rate=0.596 tok/s=158.5
explain_concept    pred= 192 draft= 340 acc= 121 rate=0.356 tok/s=109.0
summarize          pred=  52 draft=  80 acc=  37 rate=0.463 tok/s=128.5
qa_factual         pred= 192 draft= 279 acc= 135 rate=0.484 tok/s=137.3
translation        pred= 110 draft= 150 acc=  81 rate=0.540 tok/s=146.0
creative_short     pred=  34 draft=  60 acc=  23 rate=0.383 tok/s=111.1
stepwise_math      pred= 192 draft= 189 acc= 153 rate=0.809 tok/s=196.7
json_output        pred= 192 draft= 172 acc= 156 rate=0.907 tok/s=212.1
long_reasoning     pred= 192 draft= 225 acc= 145 rate=0.644 tok/s=163.1
repeat_pattern     pred= 192 draft= 159 acc= 159 rate=1.000 tok/s=234.8
code_completion    pred= 192 draft= 178 acc= 155 rate=0.871 tok/s=208.2
long_code_review   pred= 192 draft= 329 acc= 124 rate=0.377 tok/s=111.2

Aggregate:
  requests:            13
  predicted tokens:    2116
  draft tokens:        2606
  accepted tokens:     1582
  acceptance rate:     0.6071
  total wall time:     14.81 s

DFlash Comparison

A separate DFlash draft was also tested at n_max = 2:

code_python        pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=113.2
code_cpp           pred= 192 draft= 164 acc= 108 rate=0.658 tok/s=104.3
explain_concept    pred= 192 draft= 204 acc=  88 rate=0.431 tok/s=84.1
summarize          pred=  51 draft=  46 acc=  28 rate=0.609 tok/s=98.0
qa_factual         pred= 192 draft= 179 acc= 101 rate=0.564 tok/s=95.9
translation        pred=  70 draft=  78 acc=  31 rate=0.397 tok/s=80.7
creative_short     pred=  34 draft=  40 acc=  14 rate=0.350 tok/s=76.1
stepwise_math      pred= 192 draft= 151 acc= 115 rate=0.762 tok/s=113.1
json_output        pred= 192 draft= 134 acc= 124 rate=0.925 tok/s=126.4
long_reasoning     pred= 192 draft= 163 acc= 108 rate=0.663 tok/s=101.8
repeat_pattern     pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=129.2
code_completion    pred= 192 draft= 137 acc= 122 rate=0.890 tok/s=123.9
long_code_review   pred= 192 draft= 193 acc=  94 rate=0.487 tok/s=86.7

Aggregate:
  requests:            13
  predicted tokens:    2075
  draft tokens:        1769
  accepted tokens:     1175
  acceptance rate:     0.6642
  total wall time:     22.10 s
  effective throughput: 93.89 tok/s

Assistant vs. DFlash

In this benchmark, Assistant/MTP at n_max = 4 delivered approximately 52.7% higher effective aggregate throughput than the tested DFlash n_max = 2 run.

This comparison is specific to the tested draft checkpoints, quantizations, hardware, prompts, and runtime configuration. It should not be interpreted as a universal comparison between the Assistant and DFlash methods.

Observations

• n_max = 4 was the best mixed-task setting, reaching approximately 143.36 tok/s aggregate throughput.
• n_max = 5 was effectively tied overall at 142.88 tok/s, but used 15.3% more draft tokens than n_max = 4.
• Moving from n_max = 4 to n_max = 5 increased accepted draft tokens by only 2.7%, indicating sharply diminishing returns.
• n_max = 5 reached 234.8 tok/s on repeated-pattern generation.
• JSON output reached 212.1 tok/s at n_max = 5.
• Code completion reached 208.2 tok/s at n_max = 5.
• Stepwise mathematics reached 196.7 tok/s at n_max = 5.
• Open-ended explanations, creative writing, summarization, and long code review generally declined at n_max = 5.
• Assistant/MTP substantially outperformed the tested DFlash draft in this workload, especially on translation, conceptual explanation, creative writing, and long code review.

The benchmark shows a clear task-dependent trade-off: structured and predictable output benefits from deeper drafting, while high-entropy natural-language generation tends to favor shorter drafts.

Notes

• This GGUF preserves the NVFP4 expert tensors from NVIDIA's checkpoint; it is not equivalent to Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.
• The source checkpoint uses an experts-only NVFP4 recipe, so mixed tensor types in the converted GGUF are expected.
• Runtime memory requirements depend on the full stored target model, the separate Assistant model, both KV caches, and compute buffers—not only the 3.8B active parameters.
• NVIDIA officially documents vLLM on Blackwell/Linux for the source checkpoint. llama.cpp support is a separate community implementation and may behave differently.
• Native Assistant/MTP speculative decoding accelerates token generation but does not provide the same benefit to prompt prefill.
• Very short or differently terminated generations are sensitive to run-to-run variance.
• Results are specific to the tested hardware, model files, llama.cpp build, prompts, sampling configuration, and context settings.

Credits

• Google DeepMind — Gemma 4 and google/gemma-4-26B-A4B-it
• NVIDIA — ModelOpt and nvidia/Gemma-4-26B-A4B-NVFP4
• ggml-org — llama.cpp, GGUF, NVFP4 inference support, and Gemma 4 Assistant/MTP support

License

The source checkpoint is distributed under the Apache License 2.0.

Users should review the upstream nvidia/Gemma-4-26B-A4B-NVFP4 and google/gemma-4-26B-A4B-it model cards and applicable terms before redistribution or commercial use.