Gemma 4 E4B-it QAT, W4A16 with int4 per-layer embeddings, text only (unofficial)
This is an unofficial build, made and published independently of Google. It is the text-only W4A16 repack xbill9/gemma-4-E4B-it-qat-q4_0-w4a16-ct-text (revision 38b5a97) with its per-layer embedding table packed as int4. embed_tokens and lm_head stay bf16 and tied. It separates the per-layer table's effect from lm_head's: compare it with xbill9/gemma-4-E4B-it-qat-q4_0-w4a16-ct-text-emb4, which also packs embed_tokens and an untied lm_head. QAT put the embedding tables on the same 4-bit grid as the linears (group 32), so the packing recovers that grid; an off-grid group would stop the build.
| Table | bf16 | int4 |
|---|---|---|
embed_tokens_per_layer |
5.250 GiB | 1.477 GiB + f16 scales |
embed_tokens_per_layer: 0 off-grid groups; 74.78% of values bit-identical, worst error 6.62e-03 of the group max; scales 0.00155..0.938.
Checkpoint: 4.81 GiB. Linear layers are unchanged from the W4A16 repack.
Measured on one AMD Instinct MI300X
Served with stock vLLM 0.31.1rc1.dev23 (vllm/vllm-openai-rocm at sha256:ec62abecc139…) on one MI300X, 2026-10-07 to 2026-10-09, against google/gemma-4-E4B-it in bf16 on the same card and image. Output tokens per second at 1 request (128-token prompt) and 8 and 64 requests (1,024-token prompts), and the range against bf16 over nine cells of 1/8/64 requests by 128/1,024/8,192-token prompts, three repeats each:
| Weights loaded | KV cache tokens | 1 / 8 / 64 requests | vs bf16 | |
|---|---|---|---|---|
| bf16 | 14.79 GiB | 2,794,448 | 239 / 1,284 / 5,778 | 1.00x |
| This build | 5.06 GiB | 2,985,126 | 33 / 253 / 2,177 | 0.14x – 0.54x |
vLLM picked TritonW4A16LinearKernel. Accuracy was not measured on this card. Write-up: https://dev.to/gde/gemma-4-from-e2b-to-31b-on-an-amd-mi300x-fp8-overtakes-bf16-from-12b-up-h4e
Built with
embed_int4.py (in this repo, with the repack_q4_0.py helpers it imports), default fp16 scales.
Needs vLLM 0.29 or later (CompressedTensorsEmbeddingWNA16Int).
Limitations
- Text only.
- Speed measured on one AMD MI300X; accuracy not evaluated.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training and the original model card (kept unchanged in ORIGINAL_README.md) are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.