Gemma 4 12B-it QAT, FP8 W8A8, text only (unofficial)
Google's quantization-aware-trained weights, text model only, with every Linear layer stored as
FP8 E4M3 (one float32 scale per output channel) and activations quantized to FP8 per token at
run time (compressed-tensors float-quantized). Embeddings, norms and other tensors are bf16, copied byte for byte.
| FP8 Linear modules | 328 |
| Quantized values | 10,899,947,520 |
| Relative RMS error against the QAT weights | 2.64 % |
| Largest error, as a fraction of its row's largest value | 3.57 % |
| Checkpoint | 12.04 GiB |
QAT trained the weights onto a 4-bit grid with one scale per group of 32; FP8 with one scale per
output channel cannot represent those per-group scales, so this build rounds the QAT weights
again, and it also quantizes activations. Built with fp8_text.py (in this repo); no calibration data.
Limitations
- Text only.
- Unofficial. Report problems here, not to Google.