Gemma 4 26B-A4B-it QAT, FP8 E4M3 W8A8, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 26B-A4B-it weights, from google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (revision f1e06dc), text model only, with every Linear layer stored as FP8 E4M3 (one float32 scale per output channel) and activations quantized to FP8 per token at run time (compressed-tensors float-quantized). Embeddings, norms and other tensors are bf16, copied byte for byte.
The source stores each layer's 128 experts as two fused banks (experts.gate_up_proj, experts.down_proj). This build splits them into one module per expert, experts.{i}.{gate,up,down}_proj, the layout of the W4A16 repack xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct-text. The router (router.proj) stays bf16.
Measured against the QAT weights (verify_report.json) |
|
|---|---|
| FP8 Linear modules | 11,725 |
| Quantized values | 24,483,430,400 |
| Relative RMS error | 2.60 % |
| Largest error, as a fraction of its row's largest value | 3.57 % |
| Other tensors, byte-identical to their source | 392 of 392 |
| Checkpoint | 24.26 GiB |
QAT trained the weights onto a 4-bit grid with one scale per group of 32 values. FP8 with one scale per
output channel cannot represent those per-group scales, so this build rounds the QAT weights again, and it
also quantizes activations. For the exact QAT grid use xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct-text.
Measured on one AMD Instinct MI300X
Served with stock vLLM 0.31.1rc1.dev23 (vllm/vllm-openai-rocm at sha256:ec62abecc139…) on one MI300X, 2026-10-07 to 2026-10-09, against google/gemma-4-26B-A4B-it in bf16 on the same card and image. Output tokens per second at 1 request (128-token prompt) and 8 and 64 requests (1,024-token prompts), and the range against bf16 over nine cells of 1/8/64 requests by 128/1,024/8,192-token prompts, three repeats each:
| Weights loaded | KV cache tokens | 1 / 8 / 64 requests | vs bf16 | |
|---|---|---|---|---|
| bf16 | 48.54 GiB | 551,865 | 224 / 979 / 2,585 | 1.00x |
| This build | 24.72 GiB | 667,691 | 222 / 1,132 / 3,162 | 0.99x – 1.23x |
vLLM picked RowWiseTorchFP8ScaledMMLinearKernel. Accuracy was not measured on this card. Write-up: https://dev.to/gde/gemma-4-from-e2b-to-31b-on-an-amd-mi300x-fp8-overtakes-bf16-from-12b-up-h4e
Built with
fp8_text.py (in this repo),
which imports helpers from repack_q4_0.py (also here). No calibration data is used.
Limitations
- Text only.
- Speed measured on one AMD MI300X; accuracy not evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.