Gemma 4 E2B-it QAT, exact Q4_0 GGUF, text only (unofficial)

This is an unofficial build, made and published independently of Google. It is a llama.cpp GGUF of Gemma 4 E2B-it in which every 4-bit weight is the value Google's quantization-aware training (QAT) produced. Every weight matrix is stored at 4 bits, including both embedding tables. It is 2.62 GB, against 3.35 GB for Google's gemma-4-E2B-it-qat-q4_0-gguf, and follows the unquantized model much more closely. Google's model card is kept unchanged in ORIGINAL_README.md.

This is v2 (2026-09-29). v1 left per_layer_model_proj at F16 and was 2,640,154,592 B (SHA-256 25f21f14…). It remains available at revision e5d65c8. The two agree on quality within the error bars; v2 is smaller and, on a GPU, faster.

Measured with llama.cpp build f95b0d9 (318) on an i7-10750H laptop with a GTX 1650 Ti Max-Q (4 GB, no tensor cores). Divergence and perplexity: llama-perplexity on wikitext-2 test, 16 chunks of 512 tokens, against a bf16 GGUF of the same QAT weights with identical metadata; the CPU and CUDA columns share one set of bf16 base logits. Speed: llama-bench, 5 repetitions, four passes alternating which file runs first, each pass starting from the same temperature. The table gives the mean of the four passes. The same CPU divergence figures were measured independently on an i7-1360P. Logs are in evidence/v2/.

What is in it

• Language model only. Vision and audio live in Google's separate gemma-4-E2B-it-mmproj.gguf, which is not included here.
• Metadata copied byte for byte from Google's GGUF at revision 675cff4: tokenizer, chat template, hyperparameters and tensor order.
• Weights rebuilt from google/gemma-4-E2B-it-qat-q4_0-unquantized at revision 6befbac: all 275 layer matrices (attention, MLP, per-layer gate and projection), both embedding tables (token_embd, per_layer_token_embd) and per_layer_model_proj, as Q4_0.
• Copied unchanged: the norms and scale vectors (F32, 1.1 MB in all).

Why it is smaller and closer

QAT trained these weights on a 4-bit grid: in every block of 32 values, each value is a block scale times an integer from -8 to 7. That holds for the two embedding tables and per_layer_model_proj as well as the layers. The embedding tables make up 2.75 billion of the model's values, and Google's GGUF stores them as Q6_K, at 6.56 bits per weight. Stored as Q4_0 they take 4.5 bits per weight and keep the trained values, which is where most of the 0.68 GiB goes.

For the layer matrices, Google's GGUF uses llama.cpp's standard Q4_0 step, the block's largest magnitude divided by 8. QAT often puts a block's largest value on level 6 or 7, and in those blocks the standard step shifts values by up to one full level: 51% of the values in blk.0.attn_q match the trained weights. This build recovers each block's trained step instead (largest magnitude divided by 8, 7, … 1, whichever puts every value on an integer level, then refined by least squares) and stores it as the fp16 block scale.

Sampled against the bf16 QAT weights (512 rows per tensor):

Every one of the rebuilt blocks landed on a 4-bit grid, and 96.84% of rebuilt values are bit-identical to the source after bf16 rounding. The rest differ by the fp16 rounding of the scale. Per-tensor counts are in evidence/v2/build_report.json.

The generation speed-up fits a smaller set of weights read per token: token_embd doubles as the output projection, so the whole table is read for every generated token, and at Q4_0 it is 31% smaller than at Q6_K. On the GPU, v2's per_layer_model_proj at Q4_0 adds another 1.6–1.9% over v1. Memory traffic was not measured directly.

per_layer_token_embd (1.32 GB, half the file) is read a few rows per token straight from the memory-mapped file. Forcing it into VRAM with -ot 'per_layer_token_embd\.weight=CUDA0' fits on a 4 GB card and changed neither speed by more than 0.7%, so there is no reason to.

Running it

llama.cpp (tested on CPU and CUDA):

llama-server -m gemma-4-E2B-it-q4_0-exact.gguf -ngl 99 -fa 1 -c 8192 --jinja   # GPU
llama-server -m gemma-4-E2B-it-q4_0-exact.gguf -t 6 -c 8192 --jinja            # CPU

Do not pass --no-mmap: llama.cpp reads per_layer_token_embd lazily from the map.

llama-server turns thinking on by default, so a reply lands in reasoning_content and a short max_tokens can leave content empty. To get direct answers, start the server with --reasoning off, or send "chat_template_kwargs": {"enable_thinking": false} with a request. A request can also turn thinking back on with "enable_thinking": true.

Rebuilding it

hf download google/gemma-4-E2B-it-qat-q4_0-unquantized --revision 6befbac --local-dir src
hf download google/gemma-4-E2B-it-qat-q4_0-gguf gemma-4-E2B_q4_0-it.gguf --revision 675cff4 --local-dir gguf
python3 gguf_exact.py src gguf/gemma-4-E2B_q4_0-it.gguf gemma-4-E2B-it-q4_0-exact.gguf

Needs only numpy, and takes about 5 minutes on a laptop. The output is deterministic, and it was rebuilt on two different machines with the same result: SHA-256 419db9a6bf3bc15d770c85ccf9216827aa88c64fe8f3d72fbe5674f48efe2dc8. --bf16 writes the reference instead: the same file with the source bf16 weights. The script stops if any block fails to land on a 4-bit grid.

evidence/qat_grid.py and evidence/qat_lattice.py check any safetensors checkpoint for the QAT grid. On this source they also show the audio tower's attention and feed-forward weights trained at 2 bits (levels -2 to 1), and the vision tower at full precision.

Limitations

• Text only.
• Tested with llama.cpp on two laptop CPUs (AVX2) and one GPU (GTX 1650 Ti, CUDA). Other backends and runtimes were not tested.
• The divergence figures cover 8,192 tokens of English Wikipedia text.
• Unofficial. Report problems here, not to Google.

License and attribution

Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training, tokenizer, chat template and the original model card are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.