Google Gemma 4 E4B IT — NVFP4 GGUF
NVFP4 (4-bit NVIDIA FP4) GGUF of google/gemma-4-E4B-it, for NVIDIA Blackwell GPUs with native FP4 tensor core support.
Note: Access to the original model requires accepting Google's terms. By using this quantized version, you agree to the original license (Apache 2.0).
Model details
| Property | Value |
|---|---|
| Base model | google/gemma-4-E4B-it |
| Architecture | gemma4 (Gemma4ForConditionalGeneration) — 42 layers, 2560 hidden |
| Quantization | NVFP4 (general.file_type = 40, ggml type 40) |
| Parameters | 7,518,069,290 |
| BPW | 5.50 (whole file) |
| Context length | 131,072 tokens |
| Vocabulary | 262,144 tokens |
| Modalities | Text + Vision + Audio (via mmproj) |
| License | Apache 2.0 |
Tensor layout
720 tensors: 378 NVFP4 + 339 F32 (norms/biases) + 2 Q6_K (token_embd, per_layer_token_embd) + 1 F16 (per_layer_model_proj).
The two embedding matrices — including the 10,752 × 262,144 per_layer_token_embd — are deliberately kept at Q6_K rather than 4-bit. At 2.8B and 0.67B parameters they dominate any 4-bit error budget, and quantizing them to NVFP4 costs measurably more quality than it saves in size.
Chat template
Carries Google's canonical Gemma 4 template, Published: 2026-07-09 — the fix for tool-calling loops, turn closures, and thinking content-ordering. Rebuilt from google/gemma-4-E4B-it with the canonical chat_template.jinja baked in.
Files
| File | Size | Description |
|---|---|---|
gemma-4-E4B-it-NVFP4.gguf |
5,185,931,296 B (5.19 GB) | Main model weights, NVFP4 |
mmproj-gemma-4-E4B-it-f16.gguf |
990,372,608 B (990 MB) | Multimodal projector for vision/audio |
Usage
# Text-only inference
llama-cli -m gemma-4-E4B-it-NVFP4.gguf -ngl 99 --jinja
# Multimodal (image input) — --jinja is MANDATORY here
llama-mtmd-cli -m gemma-4-E4B-it-NVFP4.gguf \
--mmproj mmproj-gemma-4-E4B-it-f16.gguf \
--image path/to/image.jpg -ngl 99 --jinja
--jinja is not optional for Gemma 4. Without it, the template goes through llama.cpp's legacy converter, throws, and the process dies with 0xC0000409 right after init_audio().
Verified: loads clean and answers correctly (ftype NVFP4, exit code 0) on llama.cpp b1868 / RTX 5060 Ti 16 GB / CUDA 13.2.
Notes
- NVFP4 vs MXFP4 on Blackwell: both formats take llama.cpp's native FP4 tensor-core path (
blackwell_mma_available()gates both), so neither has a kernel advantage. NVFP4 wins on prompt processing (compute-bound, finer 16-element E4M3 scaling) while the smaller MXFP4 file wins slightly on generation (bandwidth-bound). Measured on an RTX 5060 Ti with the 12B sibling: NVFP4 up to +17% prefill, MXFP4 up to +8% decode. - This build replaces an earlier artifact that carried a stale 16,804-char pre-2026-07-09 template. The quantization scheme is unchanged (378 NVFP4 + 339 F32 + 2 Q6_K); only the template was updated. Compare against the 12B siblings NVFP4 and MXFP4 for full 128K/256K benchmark tables.
Source
- Base model: google/gemma-4-E4B-it
- Pipeline: bf16 safetensors → F16 GGUF (
convert_hf_to_gguf.py) → NVFP4 (llama-quantize) - Template source:
chat_template.jinjafrom the base repo (canonical 2026-07-09) - Sibling: FreedomAISVR/Gemma-4-E4B-it-MXFP4-GGUF — same base model in MXFP4