Gemma 4 31B-it QAT, FP8 E4M3 W8A8, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 31B-it weights, from google/gemma-4-31B-it-qat-q4_0-unquantized (revision 1e4d8be), text model only, with every Linear layer stored as FP8 E4M3 (one float32 scale per output channel) and activations quantized to FP8 per token at run time (compressed-tensors float-quantized). Embeddings, norms and other tensors are bf16, copied byte for byte.
Measured against the QAT weights (verify_report.json) |
|
|---|---|
| FP8 Linear modules | 410 |
| Quantized values | 29,286,727,680 |
| Relative RMS error | 2.64 % |
| Largest error, as a fraction of its row's largest value | 3.57 % |
| Other tensors, byte-identical to their source | 422 of 422 |
| Checkpoint | 29.92 GiB |
QAT trained the weights onto a 4-bit grid with one scale per group of 32 values. FP8 with one scale per
output channel cannot represent those per-group scales, so this build rounds the QAT weights again, and it
also quantizes activations. For the exact QAT grid use xbill9/gemma-4-31B-it-qat-q4_0-w4a16-ct-text.
Measured on one AMD Instinct MI300X
Served with stock vLLM 0.31.1rc1.dev23 (vllm/vllm-openai-rocm at sha256:ec62abecc139…) on one MI300X, 2026-10-07 to 2026-10-09, against google/gemma-4-31B-it in bf16 on the same card and image. Output tokens per second at 1 request (128-token prompt) and 8 and 64 requests (1,024-token prompts), and the range against bf16 over nine cells of 1/8/64 requests by 128/1,024/8,192-token prompts, three repeats each:
| Weights loaded | KV cache tokens | 1 / 8 / 64 requests | vs bf16 | |
|---|---|---|---|---|
| bf16 | 58.99 GiB | 121,290 | 58 / 306 / 950 | 1.00x |
| This build | 30.63 GiB | 155,730 | 71 / 432 / 1,219 | 1.20x – 1.45x |
vLLM picked RowWiseTorchFP8ScaledMMLinearKernel. Accuracy was not measured on this card. Write-up: https://dev.to/gde/gemma-4-from-e2b-to-31b-on-an-amd-mi300x-fp8-overtakes-bf16-from-12b-up-h4e
Built with
fp8_text.py (in this repo),
which imports helpers from repack_q4_0.py (also here). No calibration data is used.
Limitations
- Text only.
- Speed measured on one AMD MI300X; accuracy not evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.