Gemma 4 26B-A4B-it QAT, int8 W8A8 with int4 embeddings, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 26B-A4B-it weights, from google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (revision f1e06dc), text model only, stored as compressed-tensors int8 W8A8: int8 weights with one bf16 scale per output channel, activations quantized to int8 per token at run time (format: int-quantized). The QAT weights already sit on the Q4_0 grid; this build rounds them to int8 per channel.
embed_tokens and lm_head are the int4 tables (group 32, fp16 scales) of xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct-text-emb4 (revision 38971d3), copied unchanged; lm_head is untied. QAT put the embedding on the same 4-bit grid as the linears, so those tables keep the trained values. The other tensors are identical to xbill9/gemma-4-26B-A4B-it-qat-w8a8-int8.
The source stores each layer's 128 experts as two fused banks (experts.gate_up_proj, experts.down_proj). This build splits them into one module per expert, experts.{i}.{gate,up,down}_proj, the layout of the W4A16 repack xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct-text. The router (router.proj) stays bf16.
| Layer | Tensors | int8 against QAT, relative error |
|---|---|---|
experts.down_proj |
3,840 | 0.66 %–0.76 % (mean 0.69 %) |
experts.gate_proj |
3,840 | 0.69 %–0.98 % (mean 0.75 %) |
experts.up_proj |
3,840 | 0.56 %–0.93 % (mean 0.75 %) |
mlp.down_proj |
30 | 0.79 %–0.86 % (mean 0.82 %) |
mlp.gate_proj |
30 | 0.76 %–0.83 % (mean 0.80 %) |
mlp.up_proj |
30 | 0.76 %–0.83 % (mean 0.81 %) |
self_attn.k_proj |
30 | 0.81 %–1.29 % (mean 0.94 %) |
self_attn.o_proj |
30 | 0.84 %–1.38 % (mean 0.96 %) |
self_attn.q_proj |
30 | 0.82 %–1.12 % (mean 0.87 %) |
self_attn.v_proj |
25 | 0.80 %–1.15 % (mean 0.92 %) |
Checkpoint: 23.63 GiB, against 24.23 GiB for the build with bf16 embeddings.
Status: built and checked offline against its source; not served. The AMD MI300X sweep at this size ran bf16, q4w4a16, fp8 only. The E4B and 12B builds made by the same script stop at load on the stock ROCm vLLM image (AttributeError: 'VocabParallelEmbedding' object has no attribute 'weight'); this one was not tried.
Built with
w8a8_from_qat.py then w8a8_emb4.py (in this repo, with the repack_q4_0.py helpers they import).
No calibration data is used.
Limitations
- Text only.
- Not served or evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.