Gemma 4 12B-it QAT, FP8 E4M3 W8A8 with int4 embeddings, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 12B-it weights, from google/gemma-4-12B-it-qat-q4_0-unquantized (revision b6ed862), text model only, with every Linear layer stored as FP8 E4M3 (one float32 scale per output channel) and activations quantized to FP8 per token at run time (compressed-tensors float-quantized). Embeddings and lm_head are the int4 tables of xbill9/gemma-4-12B-it-qat-q4_0-w4a16-ct-text-emb4 (revision 94ff29e), copied unchanged; lm_head is untied.
Measured against the QAT weights (verify_report.json) |
|
|---|---|
| FP8 Linear modules | 328 |
| Quantized values | 10,899,947,520 |
| Relative RMS error | 2.64 % |
| Largest error, as a fraction of its row's largest value | 3.57 % |
| Other tensors, byte-identical to their source | 337 of 337 |
| Checkpoint | 11.22 GiB |
QAT trained the weights onto a 4-bit grid with one scale per group of 32 values. FP8 with one scale per
output channel cannot represent those per-group scales, so this build rounds the QAT weights again, and it
also quantizes activations. For the exact QAT grid use xbill9/gemma-4-12B-it-qat-q4_0-w4a16-ct-text.
Measured on one AMD Instinct MI300X
Served with stock vLLM 0.31.1rc1.dev23 (vllm/vllm-openai-rocm at sha256:ec62abecc139…) on one MI300X, 2026-10-07 to 2026-10-09, against google/gemma-4-12B-it in bf16 on the same card and image. Output tokens per second at 1 request (128-token prompt) and 8 and 64 requests (1,024-token prompts), and the range against bf16 over nine cells of 1/8/64 requests by 128/1,024/8,192-token prompts, three repeats each:
| Weights loaded | KV cache tokens | 1 / 8 / 64 requests | vs bf16 | |
|---|---|---|---|---|
| bf16 | 22.83 GiB | 439,003 | 134 / 677 / 2,248 | 1.00x |
| This build | 11.83 GiB | 474,327 | 139 / 863 / 2,725 | 1.04x – 1.29x |
vLLM picked RowWiseTorchFP8ScaledMMLinearKernel, TritonW4A16LinearKernel. Accuracy was not measured on this card. Write-up: https://dev.to/gde/gemma-4-from-e2b-to-31b-on-an-amd-mi300x-fp8-overtakes-bf16-from-12b-up-h4e
Built with
fp8_text.py (in this repo, build-on the int4-embedding build),
which imports helpers from repack_q4_0.py (also here). No calibration data is used.
Limitations
- Text only.
- Speed measured on one AMD MI300X; accuracy not evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.