Gemma 4 12B-it QAT, W4A16 with int4 embeddings, text only (unofficial)
The text-only W4A16 repack xbill9/gemma-4-12B-it-qat-q4_0-w4a16-ct-text with its embedding tables packed as int4 too, and lm_head untied and stored as the same int4 levels (vLLM ties the output layer by copying the embedding's .weight, which a packed embedding does not have). The model was trained tied.
QAT put the embedding tables on the same 4-bit grid as the linears (group 32), so the packing
recovers that grid; an off-grid group would abort the build.
| Table | bf16 | int4 |
|---|---|---|
embed_tokens |
1.875 GiB bf16 | 0.527 GiB int4 + f16 scales |
embed_tokens: 0 off-grid groups; 73.68% of values bit-identical, worst error 6.58e-03 of the group max; scales 0.000467..0.652.
Checkpoint: 6.77 GiB. Linear layers are unchanged from the W4A16 repack.
Made with embed_int4.py
(--embed-tokens, default fp16 scales). Needs vLLM 0.29 or later
(CompressedTensorsEmbeddingWNA16Int).
Limitations
- Text only.
- Unofficial. Report problems here, not to Google.