Google's gemma-4-E2B-it-qat-q4_0-unquantized checkpoint was trained so that its linear weights lie on a 4-bit grid: every block of 32 is a step times integers in -8..7, with a step of its own. Its published q4_0 GGUF quantizes those weights again with llama.cpp's rule, a step of the block's largest-magnitude value over -8. That rule misses the trained steps by about 5%, so the quantization-aware training is partly lost.
This file is Google's q4_0 GGUF, gemma-4-E2B_q4_0-it.gguf, with only the
q4_0 tensor data replaced. Each block's step is fitted to the checkpoint
by least squares, which recovers the trained grid: the relative weight
error falls from about 5% to about 0.14%, the rounding of bf16 itself.
Metadata, the tokenizer, the q6_K embeddings and the norms are Google's,
byte for byte. Only the text model is included. No vision or audio
projector.
| Model | Top-1 agreement | Mean KL |
|---|---|---|
| Google's q4_0 GGUF | 88.4% | 0.0494 |
| This GGUF | 97.2% | 0.0047 |
The table measures agreement with an int8 build of the same checkpoint over about 1,000 positions of WikiText-103 text, under teacher forcing. For scale, that int8 build agrees with the bf16 checkpoint run in Transformers on 96.5% of 2,388 positions (mean KL 0.0052).
gemma-4-E2B-it-qat-q4_0-refit.gguf: the model. 3,349,516,256 bytes,
SHA-256 ad5ae441f8efc3f5588a9567ff6cc31d99e104f94815996fc2dae0ff6d06cf28.refit.py: the script that made it, using numpy only.The layout matches Google's GGUF, so it should load wherever that one does. It was tested with a standalone C inference engine, not with llama.cpp.
python refit.py gemma-4-E2B_q4_0-it.gguf \
gemma-4-E2B-it-qat-q4_0-unquantized/model.safetensors \
-o gemma-4-E2B-it-qat-q4_0-refit.gguf
The inputs are Google's GGUF, with SHA-256 fa401b55…dec6634, and the
QAT checkpoint's model.safetensors, with SHA-256 33fe0cec…c6bf0137.
Running refit.py --check N checkpoint reports the fit for the first N
matrices without writing anything.
Gemma 4 is released by Google under the Apache License 2.0. This derivative is released under the same terms; see Google's license page.