Gemma 4 E2B-it QAT, FP8 W8A8, text only (unofficial)
Google's quantization-aware-trained gemma-4-E2B-it-qat-q4_0-unquantized weights,
text model only, with every Linear layer stored as FP8 E4M3 and activations
quantized to FP8 per token at run time (compressed-tensors float-quantized). On
GPUs with FP8 tensor cores (Ada, e.g. NVIDIA L4 and L40S; Hopper; Blackwell) vLLM
runs these layers as native FP8 matrix multiplies.
| Architecture | Gemma4ForCausalLM (gemma4_text), vision and audio towers removed |
| Linear layers | 276, FP8 E4M3 weights, one float32 scale per output channel |
| Activations | FP8, dynamic, per token |
| Embeddings, norms, per-layer embeddings | bf16, copied byte for byte (264 tensors) |
| Checkpoint | 7.38 GB |
How it differs from the W4A16 builds
QAT trained these weights onto a 4-bit grid with one scale per group of 32 values.
The W4A16 repack xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text
stores that grid exactly. FP8 with one scale per output channel cannot represent the
per-group scales, so this build rounds the QAT weights again:
Measured against the source bf16 weights (verify_report.json) |
|
|---|---|
| Relative RMS error, all 1,876,819,968 quantized values | 2.64 % |
| Largest error, as a fraction of its row's largest value | 3.57 % |
Activations are also quantized, which the W4A16 builds do not do. Use this build to measure FP8 execution on hardware that has it, not as a more exact copy of the QAT weights.
Serving
Needs a vLLM that supports compressed-tensors FP8 dynamic quantization, and an FP8 GPU for native execution:
vllm serve xbill9/gemma-4-E2B-it-qat-q4_0-fp8-text --max-model-len 8192
Built with
fp8_text.py (in this repo), from the source checkpoint above plus the config and
tensor list of the W4A16 text-only repack. No calibration data is used: weight scales
are max|row| / 448, and activation scales are computed per token at run time.
Limitations
- Text only.
- Unofficial. Report problems here, not to Google.