gemma-4-31B-it-NVFP4-GGUF

GGUF conversion of RedHatAI/gemma-4-31B-it-NVFP4 for use with llama.cpp.

This is an NVFP4-quantized standalone Gemma 4 target model.

It can run by itself. The speculative-decoding benchmark below pairs it with a compatible Gemma 4 31B Assistant/MTP draft model.

Model Details

• Source model: RedHatAI/gemma-4-31B-it-NVFP4
• Base model: google/gemma-4-31B-it
• Format: GGUF
• Quantization: NVIDIA NVFP4 / FP4
• Quantization framework: Red Hat AI LLM Compressor / compressed-tensors
• Architecture: Gemma 4 31B dense multimodal instruction model
• Input: Text and image
• Output: Text
• Purpose: Local inference and Assistant/MTP speculative decoding with llama.cpp

The upstream checkpoint quantizes the weights and activations of linear operators inside the transformer blocks to NVFP4. The vision tower, embedding layers, and output head remain at their original precision.

The upstream model card specifies:

• FP4 weight quantization with group_size = 16
• FP4 activation quantization with local per-group scaling
• Approximately 75% lower disk and GPU-memory requirements than the unquantized 16-bit checkpoint
• Release date: 2026-04-04
• Model version: 1.0

This repository contains the complete target model. The compatible Assistant/MTP draft model used for speculative decoding is separate.

Compatibility

A recent version of llama.cpp with Gemma 4, NVFP4, multimodal, and Assistant/MTP support is required.

Tested with:

• Windows
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp CUDA backend
• A compatible Gemma 4 31B Assistant GGUF draft model
• --spec-type draft-mtp

Older llama.cpp builds may fail to recognize the nvfp4 tensor type, load the Assistant architecture, or initialize the target and draft KV caches correctly.

Performance varies with the llama.cpp build, target and draft quantizations, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.

The benchmark below is text-only. Image input requires a compatible multimodal projector and llama.cpp multimodal support.

Usage

Replace the Assistant filename below with your compatible Gemma 4 31B Assistant GGUF.

llama-server

llama-server \
  -m gemma-4-31B-it-NVFP4.gguf \
  -md gemma-4-31B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 5 \
  --flash-attn on \
  -ngl 99

llama-cli

llama-cli \
  -m gemma-4-31B-it-NVFP4.gguf \
  -md gemma-4-31B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 5 \
  --flash-attn on \
  -ngl 99

Multi-GPU example

llama-server \
  -m gemma-4-31B-it-NVFP4.gguf \
  -md gemma-4-31B-it-assistant.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 5 \
  --split-mode layer \
  --tensor-split 1,1 \
  --flash-attn on \
  -ngl 99

The target and Assistant models must use compatible tokenizer, vocabulary, and Gemma 4 architecture assumptions.

An even tensor split is only a starting point. The best split depends on GPU speed, available VRAM, PCIe topology, context length, and placement of the target model, Assistant model, and KV caches.

Suggested Settings

Based on the mixed-task benchmark below:

• n_max = 5 gave the highest aggregate throughput of the tested settings.
• n_max = 2 gave the highest aggregate acceptance rate and remained the best choice for creative writing and other high-entropy output.
• n_max = 3 was a strong balanced setting with substantially less draft work than n_max = 5.
• n_max = 4 improved most structured tasks and remained a reasonable general-purpose compromise.
• n_max = 5 was best for code, JSON, repeated patterns, mathematical reasoning, and long reasoning.
• Creative writing degraded steadily as draft depth increased.

Recommended maximum-throughput setting:

--spec-draft-n-max 5

Recommended balanced setting:

--spec-draft-n-max 3

For role-play, creative writing, open-ended conversation, or other high-entropy generation:

--spec-draft-n-max 2

Conversion

Converted from the original Hugging Face compressed-tensors checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:

python convert_hf_to_gguf.py \
  /path/to/gemma-4-31B-it-NVFP4 \
  --outfile gemma-4-31B-it-NVFP4.gguf \
  --outtype auto \
  --verbose

Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors.

The conversion repackages the compressed-tensors NVFP4 checkpoint into a GGUF-compatible representation rather than requantizing it into a K-quant.

The compatible Assistant model must be converted separately. No target-model weights were retrained by this conversion.

File Size

The converted GGUF may differ in size from the original Hugging Face checkpoint.

The conversion does not copy the original safetensors byte-for-byte. NVFP4 tensors are repacked into GGML-compatible layouts, and GGUF stores the scale metadata required by llama.cpp. The vision tower, embeddings, output head, and other tensors that remain at higher precision also contribute to the file size.

A larger GGUF does not mean the model was converted to FP16 or to a conventional Q4 format.

Windows reports file sizes using binary units even though File Explorer labels them as GB. The same file can therefore appear smaller in Windows than on Hugging Face.

File size is not the same as runtime VRAM usage. Additional memory is needed for:

• the Assistant draft model
• target and draft KV caches
• CUDA context
• compute buffers
• graph workspace
• multimodal projector, when used

Verifying the GGUF

The converted target can be checked with llama-gguf:

.\llama-gguf.exe E:\HF_MODELS\gemma-4-31B-it-NVFP4.gguf r

A successful conversion should show NVFP4 transformer tensors and their associated scale tensors. The vision tower, embeddings, and output head may remain in higher-precision formats.

Example:

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.

Benchmark

Benchmarked using mtp-bench with a compatible Gemma 4 31B Assistant model and:

--spec-type draft-mtp

Aggregate Results — Assistant/MTP

Effective batch throughput is calculated as total predicted tokens divided by total wall time.

Per-request tok/s can be higher because aggregate wall time may include prompt processing, request transitions, and benchmark overhead.

The predicted-token totals differ slightly because some generated responses terminated at different lengths. The n_max = 4 and n_max = 5 runs both generated 2,016 tokens and can be compared directly.

These tests do not include a non-speculative baseline and therefore should not be interpreted as direct speedup ratios over standard decoding.

Detailed Assistant Results

n_max = 2

code_python        pred= 192 draft= 138 acc= 121 rate=0.877 tok/s=61.6
code_cpp           pred= 192 draft= 147 acc= 117 rate=0.796 tok/s=58.4
explain_concept    pred= 192 draft= 170 acc= 105 rate=0.618 tok/s=50.2
summarize          pred=  48 draft=  38 acc=  30 rate=0.789 tok/s=56.5
qa_factual         pred= 172 draft= 146 acc=  99 rate=0.678 tok/s=53.2
translation        pred=  16 draft=  12 acc=  10 rate=0.833 tok/s=57.5
creative_short     pred=  34 draft=  38 acc=  14 rate=0.368 tok/s=39.7
stepwise_math      pred= 192 draft= 138 acc= 122 rate=0.884 tok/s=62.8
json_output        pred= 192 draft= 129 acc= 126 rate=0.977 tok/s=65.7
long_reasoning     pred= 192 draft= 147 acc= 117 rate=0.796 tok/s=58.3
repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=67.7
code_completion    pred= 173 draft= 126 acc= 110 rate=0.873 tok/s=61.7
long_code_review   pred= 192 draft= 178 acc= 101 rate=0.567 tok/s=46.4

Aggregate:
  requests:            13
  predicted tokens:    1979
  draft tokens:        1534
  accepted tokens:     1199
  acceptance rate:     0.7816
  total wall time:     37.27 s

n_max = 3

code_python        pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=71.7
code_cpp           pred= 192 draft= 178 acc= 131 rate=0.736 tok/s=67.3
explain_concept    pred= 192 draft= 225 acc= 116 rate=0.516 tok/s=53.7
summarize          pred=  48 draft=  48 acc=  33 rate=0.688 tok/s=62.6
qa_factual         pred= 172 draft= 171 acc= 114 rate=0.667 tok/s=63.3
translation        pred=  16 draft=  15 acc=  11 rate=0.733 tok/s=64.1
creative_short     pred=  34 draft=  54 acc=  16 rate=0.296 tok/s=39.7
stepwise_math      pred= 192 draft= 168 acc= 134 rate=0.798 tok/s=70.5
json_output        pred= 192 draft= 144 acc= 142 rate=0.986 tok/s=80.7
long_reasoning     pred= 192 draft= 177 acc= 131 rate=0.740 tok/s=65.8
repeat_pattern     pred= 192 draft= 143 acc= 143 rate=1.000 tok/s=83.8
code_completion    pred= 173 draft= 147 acc= 124 rate=0.844 tok/s=73.5
long_code_review   pred= 192 draft= 236 acc= 111 rate=0.470 tok/s=48.6

Aggregate:
  requests:            13
  predicted tokens:    1979
  draft tokens:        1874
  accepted tokens:     1341
  acceptance rate:     0.7156
  total wall time:     32.96 s

n_max = 4

code_python        pred= 192 draft= 199 acc= 141 rate=0.709 tok/s=77.7
code_cpp           pred= 192 draft= 201 acc= 140 rate=0.697 tok/s=77.3
explain_concept    pred= 192 draft= 273 acc= 122 rate=0.447 tok/s=57.5
summarize          pred=  48 draft=  56 acc=  33 rate=0.589 tok/s=70.0
qa_factual         pred= 192 draft= 229 acc= 133 rate=0.581 tok/s=66.8
translation        pred=  16 draft=  16 acc=  11 rate=0.688 tok/s=71.3
creative_short     pred=  32 draft=  64 acc=  16 rate=0.250 tok/s=39.0
stepwise_math      pred= 192 draft= 189 acc= 143 rate=0.757 tok/s=77.1
json_output        pred= 192 draft= 157 acc= 151 rate=0.962 tok/s=91.2
long_reasoning     pred= 192 draft= 213 acc= 137 rate=0.643 tok/s=67.9
repeat_pattern     pred= 192 draft= 155 acc= 152 rate=0.981 tok/s=94.2
code_completion    pred= 192 draft= 168 acc= 149 rate=0.887 tok/s=86.5
long_code_review   pred= 192 draft= 286 acc= 119 rate=0.416 tok/s=49.0

Aggregate:
  requests:            13
  predicted tokens:    2016
  draft tokens:        2206
  accepted tokens:     1447
  acceptance rate:     0.6559
  total wall time:     31.35 s

n_max = 5

code_python        pred= 192 draft= 219 acc= 147 rate=0.671 tok/s=83.2
code_cpp           pred= 192 draft= 235 acc= 144 rate=0.613 tok/s=78.4
explain_concept    pred= 192 draft= 331 acc= 124 rate=0.375 tok/s=53.2
summarize          pred=  48 draft=  60 acc=  37 rate=0.617 tok/s=76.0
qa_factual         pred= 192 draft= 270 acc= 136 rate=0.504 tok/s=63.2
translation        pred=  16 draft=  20 acc=  12 rate=0.600 tok/s=66.7
creative_short     pred=  32 draft=  80 acc=  16 rate=0.200 tok/s=35.8
stepwise_math      pred= 192 draft= 200 acc= 150 rate=0.750 tok/s=83.1
json_output        pred= 192 draft= 169 acc= 157 rate=0.929 tok/s=100.6
long_reasoning     pred= 192 draft= 238 acc= 142 rate=0.597 tok/s=74.8
repeat_pattern     pred= 192 draft= 165 acc= 158 rate=0.958 tok/s=104.6
code_completion    pred= 192 draft= 183 acc= 154 rate=0.842 tok/s=97.9
long_code_review   pred= 192 draft= 325 acc= 124 rate=0.382 tok/s=52.8

Aggregate:
  requests:            13
  predicted tokens:    2016
  draft tokens:        2495
  accepted tokens:     1501
  acceptance rate:     0.6016
  total wall time:     30.18 s

Observations

• n_max = 5 delivered the highest effective aggregate throughput at approximately 66.80 tok/s.
• Relative to n_max = 2, n_max = 5 improved aggregate throughput by approximately 25.8%.
• Relative to the previously tested lower-quality Assistant pairing at n_max = 2, the final Assistant pairing and n_max = 5 improved throughput from approximately 48.60 tok/s to 66.80 tok/s.
• n_max = 5 reached 104.6 tok/s on repeated-pattern generation.
• JSON output reached 100.6 tok/s.
• Code completion reached 97.9 tok/s.
• Code and stepwise mathematics exceeded 78 tok/s.
• Creative writing did not benefit from deeper drafting and declined from 39.7 tok/s at n_max = 2 to 35.8 tok/s at n_max = 5.
• Open-ended explanation, factual QA, and translation showed diminishing returns or regressions at n_max = 5.
• Increasing from n_max = 4 to n_max = 5 improved aggregate throughput by approximately 3.9%, while aggregate acceptance fell from 65.6% to 60.2%.

The benchmark shows a clear task-dependent trade-off: structured and predictable output benefits from deeper drafting, while high-entropy natural-language generation generally favors a shorter draft.

Upstream Accuracy

The upstream Red Hat AI model card reports strong recovery relative to the unquantized google/gemma-4-31B-it, including approximately:

• 99.0–99.6% recovery on IFEval
• 99.9% on GSM8K Platinum
• 98.9% on MMLU-Pro
• 95.2% on MATH-500
• 98.7% on AIME 2025
• 98.9% on GPQA Diamond
• 98.9% on LiveCodeBench v6

These are upstream vLLM evaluation results and are separate from the local llama.cpp throughput benchmark above.

Notes

• This GGUF preserves the NVFP4 transformer tensors from Red Hat AI's compressed-tensors checkpoint; it is not equivalent to Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.
• Mixed tensor types are expected because the vision tower, embeddings, and output head are not quantized to NVFP4.
• Runtime memory requirements depend on the full target model, the separate Assistant model, both KV caches, multimodal components, and compute buffers.
• The upstream checkpoint targets vLLM. llama.cpp support is a separate community implementation and may behave differently.
• Assistant/MTP speculative decoding accelerates token generation but does not provide the same benefit to prompt prefill.
• Very short generations, especially the translation test in these runs, are sensitive to run-to-run variance.
• Results are specific to the tested hardware, model files, llama.cpp build, prompts, sampling configuration, and context settings.

Credits

• Google DeepMind — Gemma 4 and google/gemma-4-31B-it
• Red Hat AI — NVFP4 quantization, LLM Compressor, and RedHatAI/gemma-4-31B-it-NVFP4
• ggml-org — llama.cpp, GGUF, NVFP4 inference support, and Gemma 4 Assistant/MTP support

License

The source checkpoint is distributed under the Apache License 2.0.

Users should review the upstream RedHatAI/gemma-4-31B-it-NVFP4 and google/gemma-4-31B-it model cards and applicable terms before redistribution or commercial use.