Gemma 4 E2B-it QAT, FP8 E4M3FNUZ W8A8 with int4 embeddings, text only (unofficial)
This is an unofficial build, made and published independently of Google. It holds Google's quantization-aware-trained (QAT) Gemma 4 E2B-it weights, from google/gemma-4-E2B-it-qat-q4_0-unquantized (revision 6befbac), text model only, with every Linear layer stored as FP8 E4M3FNUZ (one float32 scale per output channel) and activations quantized to FP8 per token at run time (compressed-tensors float-quantized). Embeddings, per-layer embeddings and lm_head are the int4 tables of xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 (revision db715f8), copied unchanged; lm_head is untied.
E4M3FNUZ is the FP8 format of AMD CDNA 3 (Instinct MI300X, MI300A, MI325X). It has the same 3-bit mantissa as the E4M3 used by NVIDIA, with an exponent bias one higher, so its largest value is 240 against 448, and it has no negative zero. This build rounds the QAT weights straight to that grid, with scale max|row| / 240. Its twin xbill9/gemma-4-E2B-it-qat-q4_0-fp8-text-emb4 holds the same weights in E4M3; the two builds differ by where each rounds. Against the QAT weights both have a relative RMS error of 2.64 %.
Measured against the QAT weights (verify_report.json) |
|
|---|---|
| FP8 Linear modules | 276 |
| Quantized values | 1,876,819,968 |
| Relative RMS error | 2.64 % |
| Largest error, as a fraction of its row's largest value | 3.33 % |
| Other tensors, byte-identical to their source | 262 of 262 |
| Checkpoint | 3.40 GiB |
QAT trained the weights onto a 4-bit grid with one scale per group of 32 values. FP8 with one scale per
output channel cannot represent those per-group scales, so this build rounds the QAT weights again, and it
also quantizes activations. For the exact QAT grid use xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text.
Measured on one AMD Instinct MI300X
Served with stock vLLM 0.31.1rc1.dev23 (vllm/vllm-openai-rocm at sha256:ec62abecc139…) on one MI300X, 2026-10-07 to 2026-10-09, against google/gemma-4-E2B-it in bf16 on the same card and image. Output tokens per second at 1 request (128-token prompt) and 8 and 64 requests (1,024-token prompts), and the range against bf16 over nine cells of 1/8/64 requests by 128/1,024/8,192-token prompts, three repeats each:
| Weights loaded | KV cache tokens | 1 / 8 / 64 requests | vs bf16 | |
|---|---|---|---|---|
| bf16 | 9.42 GiB | 9,045,060 | 343 / 1,795 / 8,951 | 1.00x |
| This build | 3.61 GiB | 9,422,004 | 247 / 1,752 / 9,500 | 0.72x – 1.06x |
vLLM picked RowWiseTorchFP8ScaledMMLinearKernel, TritonW4A16LinearKernel. Accuracy was not measured on this card. Write-up: https://dev.to/gde/gemma-4-e2b-on-an-amd-mi300x-which-weight-format-should-you-serve-1a01
Against its E4M3 twin on the same card it loads the same kernel and memory and runs at 0.998x to 1.006x in eight of nine cells; the ninth, 64 requests with 8,192-token prompts, varied by up to 15.7% between repeats.
Serving
On ROCm, vLLM's compressed-tensors loader casts these weights into an E4M3 parameter and then converts them to E4M3FNUZ for the GPU, doubling the scale. Every value of this checkpoint up to 240 survives that round trip except the smallest E4M3FNUZ subnormal (2^-10), which E4M3 cannot hold and rounds. Other GPUs are untested.
Built with
fp8_text.py (in this repo, with --fnuz, build-on the int4-embedding build),
which imports helpers from repack_q4_0.py (also here). No calibration data is used.
Limitations
- Text only.
- Speed measured on one AMD MI300X; accuracy not evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.