Gemma 4 31B-it QAT, int8 W8A8 with int4 embeddings, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 31B-it weights, from google/gemma-4-31B-it-qat-q4_0-unquantized (revision 1e4d8be), text model only, stored as compressed-tensors int8 W8A8: int8 weights with one bf16 scale per output channel, activations quantized to int8 per token at run time (format: int-quantized). The QAT weights already sit on the Q4_0 grid; this build rounds them to int8 per channel.
embed_tokens and lm_head are the int4 tables (group 32, fp16 scales) of xbill9/gemma-4-31B-it-qat-q4_0-w4a16-ct-text-emb4 (revision c8e1d82), copied unchanged; lm_head is untied. QAT put the embedding on the same 4-bit grid as the linears, so those tables keep the trained values. The other tensors are identical to xbill9/gemma-4-31B-it-qat-w8a8-int8.
| Layer | Tensors | int8 against QAT, relative error |
|---|---|---|
mlp.down_proj |
60 | 0.93 %–1.18 % (mean 1.02 %) |
mlp.gate_proj |
60 | 0.80 %–0.97 % (mean 0.87 %) |
mlp.up_proj |
60 | 0.80 %–0.99 % (mean 0.85 %) |
self_attn.k_proj |
60 | 0.83 %–1.35 % (mean 0.97 %) |
self_attn.o_proj |
60 | 0.82 %–1.16 % (mean 0.98 %) |
self_attn.q_proj |
60 | 0.83 %–1.07 % (mean 0.92 %) |
self_attn.v_proj |
50 | 0.77 %–1.17 % (mean 0.93 %) |
Checkpoint: 28.76 GiB, against 29.91 GiB for the build with bf16 embeddings.
Status: built and checked offline against its source; not served. The AMD MI300X sweep at this size ran bf16, q4w4a16, fp8 only. The E4B and 12B builds made by the same script stop at load on the stock ROCm vLLM image (AttributeError: 'VocabParallelEmbedding' object has no attribute 'weight'); this one was not tried.
Built with
w8a8_from_qat.py then w8a8_emb4.py (in this repo, with the repack_q4_0.py helpers they import).
No calibration data is used.
Limitations
- Text only.
- Not served or evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.