Gemma 4 31B-it QAT, compressed-tensors W4A16 (unofficial repack)
This is an unofficial repack, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) weights for Gemma 4 31B-it, from google/gemma-4-31B-it-qat-q4_0-unquantized (revision 1e4d8be), repacked into the compressed-tensors W4A16 format that vLLM loads, with the vision tower kept. The text-only build is xbill9/gemma-4-31B-it-qat-q4_0-w4a16-ct-text.
What it is
- Format: compressed-tensors
pack-quantized, symmetric int4, group size 32, bf16 scales, activations unquantized (W4A16). - Quantized: every attention and MLP projection of the language model.
- Kept bf16, copied byte for byte: embeddings, norms and the vision tower (778 of 778 tensors byte-identical).
- Size: 19.04 GiB, against 58.25 GiB for the bf16 source.
How it was made
The -qat-q4_0-unquantized export stores bf16 values that already lie on a 4-bit grid: within every group of 32
weights, each weight is step × level with level from −8 to 7. The repack recovers each group's step (the
max|w| / m, m from 1 to 8, that reproduces all 32 values, refined by least squares) and writes the levels as
packed int4. It does no new quantization.
Verification
repack_q4_0.py verify rereads both checkpoints and checks every group:
| Layer | Tensors | Groups of 32 | Levels off the source grid | Values bit-identical |
|---|---|---|---|---|
mlp.down_proj |
60 | 216,760,320 | 0 | 90.0 % |
mlp.gate_proj |
60 | 216,760,320 | 0 | 89.9 % |
mlp.up_proj |
60 | 216,760,320 | 0 | 90.0 % |
self_attn.k_proj |
60 | 37,847,040 | 0 | 90.0 % |
self_attn.o_proj |
60 | 96,337,920 | 0 | 89.7 % |
self_attn.q_proj |
60 | 96,337,920 | 0 | 89.8 % |
self_attn.v_proj |
50 | 34,406,400 | 0 | 90.3 % |
| All | 410 | 915,210,240 | 0 | 89.9 % |
Values that are not bit-identical differ through the bf16 scale, by at most 1.1e-02 relative. repack_report.json and verify_report.json in this repository are that run's outputs.
Status: built and checked offline against its source; not yet served or evaluated.
Built with
repack_q4_0.py (in this repo): repack, then verify.
Limitations
- Not yet served or evaluated; image input in particular is untested.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.