google/gemma-4-12B-it as llama.cpp GGUF reference files, for comparison with the Tenstorrent Tensix exports of the same model.
Four preserved text-decoder GGUF quantizations of Gemma 4 12B Unified, converted and quantized by Xylolabs with llama.cpp. Choose imatrix Q3_K_M, IQ4_XS with or without the existing importance matrix, or Q5_K_M.
Each alternative contains 667 tensors from all 48 text-decoder layers, the tokenizer and original chat template. Vision/audio encoders are omitted. Use one checkpoint at a time.
| File | Download GB |
|---|---|
gemma-4-12B-it-Q3_K_M-imat.gguf |
6.087 |
gemma-4-12B-it-IQ4_XS-imat.gguf |
6.635 |
gemma-4-12B-it-IQ4_XS.gguf |
6.691 |
gemma-4-12B-it-Q5_K_M.gguf |
8.547 |
Average bits include every stored tensor. The shared embedding/output matrix is Q6_K in all four files, and norms/RoPE constants retain float32 values; these higher-precision tensors raise the average above the preset name. Actual tensor types are:
| Choice | Tensor types and counts |
|---|---|
Q3_K_M-imat |
F32: 338, Q3_K: 192, Q4_K: 131, Q5_K: 5, Q6_K: 1 |
IQ4_XS-imat |
F32: 338, IQ4_XS: 328, Q6_K: 1 |
IQ4_XS |
F32: 338, IQ4_XS: 322, Q5_K: 6, Q6_K: 1 |
Q5_K_M |
F32: 338, Q5_K: 284, Q6_K: 45 |
The -imat files retain existing importance-matrix results. Historical relative calibration filenames are
metadata labels; no calibration dataset is included.
With a llama.cpp build, download one file and run:
llama-cli -m gemma-4-12B-it-Q5_K_M.gguf -c 2048 -n 32 --temp 0 --jinja \
--chat-template-kwargs '{"enable_thinking":false}' \
--single-turn -p "Name the capital of France. Reply with only the city."
This command is checked with llama.cpp v0.9.0. Use one quantization at a time. The example explicitly limits context to 2,048 tokens; original context metadata is unchanged. KV and working buffers require additional memory.
All four files pass the relative reference check: 33/33 answers plus 7/7 thinking-on rechecks, with zero counted losses or broken answers. Stored BF16 and current Q4_K_M also answer 33/33 plus 7/7.
| Choice | Mean KL | KL / Q4_K_M | Top-1 agreement |
|---|---|---|---|
Q3_K_M-imat |
0.999333 | 1.134× | 68.10% |
IQ4_XS-imat |
0.695671 | 0.789× | 73.42% |
IQ4_XS |
0.983883 | 1.116× | 68.49% |
Q5_K_M |
0.367732 | 0.417× | 80.96% |
The 16 matched 1,024-token sequences cover 16,232 scored positions across English/Korean plain/chat text groups. KL and top-1 agreement are measured against the stored BF16 pass. Current Q4_K_M has mean KL 0.881231 and 69.79% top-1 agreement. Every ratio above is below 1.5×; the answer gate allows at most one more counted lost/broken answer than Q4_K_M.
Current checks use an 8,192-token context, with all 33 prompt token sequences, seven reasoning rechecks, stop IDs and generation budgets matched to the retained reference. Near-tie exclusions apply to lost-answer comparisons (BF16 top-two gap below 1.0 at the first divergent token), as do reasoning items recovered with thinking on. Broken answers count where BF16 is correct.
The exact Q5_K_M CLI example loads the released file and completes one text query.
Apache 2.0, as declared by the original Unified model card and
its Google license link. Full standard terms are in LICENSE, and
NOTICE identifies the source and text conversion. Provided as is, without warranty or upstream endorsement. Not affiliated with or endorsed by Tenstorrent.
Cite the original Gemma 4 12B Unified model card and its linked Gemma 4 technical report.
google/gemma-4-12B-it