Gemma-4-31B-IT-NVFP4-GGUF

A family of five GGUF files of nvidia/Gemma-4-31B-IT-NVFP4, NVIDIA's 31B dense instruction-tuned model (60 attention blocks, 262,144-token native context) with native NVFP4 quantization (W4A16: 4-bit weights, FP8 block scales, group size 16). NVFP4 runs natively on Blackwell GPUs.

My part here is only the numerics: I converted the official NVFP4 checkpoint to GGUF and built a size and precision ladder for the tensors that most affect output quality and decode speed. All credit for the model itself belongs upstream (full chain below).

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.

🌐 esatapedico.com. Updates, work-in-progress, and ways to support.

The five files

Four tiers share a byte-identical 410-tensor native-NVFP4 backbone (180 feed-forward + 230 attention tensors), differing only in embedding-head precision. HIGHEST keeps the same feed-forward backbone with attention at Q8_0 instead:

Sizes are decimal GB as shown in the HF file browser. Embeddings are tied (no separate output head), so token_embd is the single head tensor. Picking a tier: HIGH is the accuracy pick — highest measured MMLU (see below), though all five sit inside a ±~1pp jitter band so the ordering is not firm. HIGHEST is the max-fidelity pick (Q8_0 attention) but does not measurably lead on task accuracy. MID-HIGH is the compact pick (full NVFP4 backbone, best speed-per-GB). All tiers need two 16 GB GPUs with a tensor split, or partial CPU offload; HIGHEST wants about 22 GiB VRAM plus KV cache.

Tensor layout

Why only token_embd varies (and what HIGHEST changes)

Embeddings are tied in this model (no separate output head), so token_embd is the only head tensor — and the only weight the official file leaves outside NVFP4. Everything else that matters is already NVFP4, and llama-quantize cannot re-quantize NVFP4 tensors, so LOW–HIGH move exactly one tensor (Q4_K → Q6_K → Q8_0 → BF16) while the backbone stays bit-identical. That is why the four tiers sit within 2 GB of each other. HIGHEST is the one tier that touches the backbone: attention goes from NVFP4 to Q8_0 (230 tensors), which is also why it is the only tier that moves task accuracy instead of just file size. Text-only source: no vision tower, no mmproj, no MTP drafter.

How this was made

At a high level, the steps were:

1. Converted the official NVFP4 checkpoint to GGUF (--outtype auto, preserving native NVFP4 tensors end to end).
2. Verified the NVFP4 tensor census (410 backbone tensors on LOW–HIGH).
3. Built each tier over the shared backbone, varying only embedding-head precision (LOW–HIGH) or attention precision plus heads (HIGHEST).
4. Verified per-tier tensor counts and backbone identity, and patched GGUF KV for name, description and license without changing tensor data.

Benchmarks (naive, single run, not comparable across setups)

These are rough sanity checks to confirm the files load and generate, not a formal benchmark. Method and hardware are noted so you can interpret them in context. Results will vary with hardware, sampling and context length. All runs used direct GPU via llama.cpp tools on dual RTX 5070 Ti (2x16 GB) with tensor split.

Smoke: 5 of 5 coherent

Prompt What is a black hole? via single-turn sampling (temp 0.8, seed 42, 220 tokens): each tier produces a natural coherent completion with no repetition or truncation. All tiers passed.

Perplexity (PPL)

Direct llama-perplexity on a 75 kB diverse English payload, same chunks for all tiers. Note: PPL here does not track task accuracy (the highest-PPL shipped tier ranks fourth of five on MMLU below) — treat it as a load/generation sanity check only:

Speed (llama-bench, pp512 prompt processing, tg128 generation, tok/s)

Task accuracy (MMLU, same 1000-question subset, seed 42)

Server-side multiple-choice eval, thinking off. Measured 2026-10-29/30, identical harness for all five tiers (ctx 8192, one slot — no context-length exclusions):

Unresolved items are single-question answer-extraction misses, not timeouts. GSM8K (200 cases): LOW 82.0%, MEDIUM 80.0%, MID-HIGH 82.0%, HIGH 80.0%, HIGHEST 80.5%. IFEval cap (30 cases): 30/30 parsed on every tier. GPQA and KLD-vs-anchor were not measured.

Running it

llama.cpp / llama-server

llama-server -m Gemma-4-31B-IT-NVFP4-HIGHEST.gguf \
  --alias gemma-4-31b -ngl all -c 8192 -fa on --jinja --port 8080

• Requires a recent llama.cpp with NVFP4 (GGML type 40) CUDA kernels and sm_120 support (Blackwell), including Gemma4 architecture support.
• Split across two 16 GB cards happens automatically (no --tensor-split needed; VMM spans both). Lower tiers fit the same way with more headroom.
• For long contexts, budget KV cache accordingly: quantize KV with -ctk q4_0 -ctv q4_0 if VRAM is tight (measured below at 8192 ctx; scale up for 262K).
• Sampling used for the smoke checks below: temp 0.8, seed 42. Task evals ran thinking off (MMLU) and thinking on (GSM8K).

Memory footprint

Measured resident VRAM during llama-bench (dual 16 GB GPUs, tensor split, 8192 ctx):

• Weights: 17.29 / 17.65 / 17.99 / 19.31 / 23.55 GB per tier (LOW → HIGHEST)
• Resident at bench: 16.09 / 16.42 / 16.74 / 17.97 / 21.92 GiB
• HIGHEST plus KV is the tightest fit: comfortable on 2x16 GB at 8192 ctx, quantize KV or shorten context for headroom.

Notes

• MMLU lands at 76.3–78.1% across tiers on the same 1000-question subset — solid for a 31B quant but below the 80%+ band of larger or full-precision models. The shortfall is in the model, not the quantization: the official 31.5 GB file scores 72.70% on the same subset.
• About 3–5% of MMLU items per run miss answer extraction (HIGHEST 12.5% in this run; fast single-question parse misses, spread across subjects, not timeouts) and count as wrong. Tighter answer parsing would add low-single-digit points to every leg equally.
• PPL on generic web text is in the hundreds-to-thousands and does not track task accuracy (the best-MMLU tier has the worst PPL). Use it as a load/generation sanity check, not a quality ranking.

Attribution & provenance

This is a derivative work built entirely from existing Apache 2.0 artifacts. Nothing here was trained or fine tuned. Credit belongs to:

1. Google for the Gemma model family (open weights).
2. NVIDIA for the official NVFP4 checkpoint, nvidia/Gemma-4-31B-IT-NVFP4 (Apache 2.0): the ModelOpt NVFP4 quantization this family converts.
3. This repo's author for the GGUF conversion and the tier ladder only.

Repository contents

• Five tier GGUFs (table above, 17.29 to 23.55 GB) plus 5 override maps (overrides-{LOW,MEDIUM,MID-HIGH,HIGH,HIGHEST}.txt, 1,193 lines each: per-tensor target types, including token_embd.weight per tier).
• The official 31.5 GB converted ORIG is the build source and is not shipped.

SHA-256

8dbcffa4d128411a6c26105df30fdbbf8fd51a8845b00d1e846e099068a4faab  Gemma-4-31B-IT-NVFP4-LOW.gguf
4fd362b74698d65534dc1fd6fe18c2981853140ddbaaf8d39697dfa8f63058fc  Gemma-4-31B-IT-NVFP4-MEDIUM.gguf
17fe15e0e677ac2a12b2239d958c759a19fd72e4012910318c9c1cd9e90765ed  Gemma-4-31B-IT-NVFP4-MID-HIGH.gguf
4bcf5f38fa8c0371109cb744b36c344f976b6f2d6d509c5b151e2aff140aace3  Gemma-4-31B-IT-NVFP4-HIGH.gguf
32c362793e2f684ad1730bbf01b26ec9772ed9b9ae86846382553d1c7657762b  Gemma-4-31B-IT-NVFP4-HIGHEST.gguf

License

Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to Google (Gemma) and NVIDIA (NVFP4 quantization) as above.

"Gemma" is a trademark of Google. Trademarks are used here only to identify upstream models; this repository is not affiliated with, sponsored by, or endorsed by Google or NVIDIA.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.