Google Gemma 4 31B IT — NVFP4 GGUF

This repository contains a NVFP4 (4-bit NVIDIA FP4) quantized GGUF of google/gemma-4-31B-it, optimized for efficient inference on NVIDIA GPUs with FP4 tensor core support.

Note: Access to the original model requires accepting Google's terms. By using this quantized version, you agree to the original license (Apache 2.0).

Model Details

Property	Value
Base Model	google/gemma-4-31B-it
Architecture	gemma4
Quantization	NVFP4 (4-bit NVIDIA FP4, ftype=39)
Parameters	~31B
Context Length	262K tokens
Vocabulary	262K tokens
Modalities	Text + Vision + Audio (via mmproj)
License	Apache 2.0

Files

File	Size	Description
`gemma-4-31B-it-NVFP4.gguf`	~16.5 GB	Main model weights (NVFP4 quantized)
`mmproj-gemma-4-31B-it-f16.gguf`	~1.2 GB	Multimodal projector for vision/audio inputs

Usage

With llama.cpp

# Text-only inference
./llama-cli -m gemma-4-31B-it-NVFP4.gguf -p "Your prompt here" -n 512

# Multimodal (image input)
./llama-cli -m gemma-4-31B-it-NVFP4.gguf --mmproj mmproj-gemma-4-31B-it-f16.gguf \
    --image path/to/image.jpg -p "Describe this image" -n 256

With LM Studio

Download all .gguf files from this repository
Place them in a folder (e.g., models/Gemma-4-31B-it-NVFP4/)
Launch LM Studio — it should auto-detect the model
The mmproj file will be automatically picked up for multimodal support

With other GGUF backends

Any GGUF-compatible backend that supports NVFP4 (ftype 39) and the gemma4 architecture can load this model.

Quantization Details

This model uses NVFP4 (NVIDIA FP4) quantization, which stores weights in 4-bit floating-point format (2:1:1 sign-exponent-mantissa). This is the same format supported natively by NVIDIA Blackwell GPUs with FP4 tensor cores.

ftype: 39 (LLAMA_FTYPE_MOSTLY_NVFP4)
Per-tensor dtype: 40 (GGML_TYPE_NVFP4)
Compression ratio: ~4× vs FP16
Requires: CUDA-capable GPU, or any backend supporting NVFP4

Multimodal Support

Gemma 4 is a natively multimodal model supporting:

Text — standard text generation
Vision — image understanding (via mmproj projector)
Audio — audio understanding (via mmproj projector)

The mmproj-gemma-4-31B-it-f16.gguf file contains the multimodal projector weights needed for vision and audio inputs.

Original Model

For full details on capabilities, training, and evaluation, see the original model card.

License

This quantized model is distributed under Apache 2.0, consistent with the original Gemma 4 license.

FreedomAISVR/Gemma-4-31B-it-NVFP4-GGUF