Gemma 4 E4B-it QAT, int8 W8A8 with int4 embeddings, text only (unofficial)

This is an unofficial build, made and published independently of Google. It is Google's quantization-aware-trained (QAT) Gemma 4 E4B-it, from google/gemma-4-E4B-it-qat-q4_0-unquantized, with int8 W8A8 linear layers and int4 token embeddings, per-layer embeddings and output head, loadable as a text-only Gemma4ForCausalLM. Google's model card is kept unchanged in ORIGINAL_README.md.

It is xbill9/gemma-4-E4B-it-qat-w8a8-int8 with the vocabulary tables made int4: the int8 linears are byte for byte the same.

What it is

• Linear layers: compressed-tensors int8 W8A8: int8 weights with one bf16 scale per output channel, activations quantized to int8 per token at run time. Rounded from the QAT weights, which already sit on the Q4_0 grid (0.60–1.70% relative error per tensor; the table is in the -w8a8-int8 card).
• embed_tokens, embed_tokens_per_layer and lm_head: pack-quantized symmetric int4, group 32, fp16 scales. The QAT weights put these tables on the same 4-bit grid as the linears, so each group's step is recovered, not chosen. lm_head is an untied copy of the embedding values (tie_word_embeddings: false), so the output layer runs as an int4 linear.
• Kept bf16: norms and scalars.
• Dropped: the vision and audio towers.
• Size: 5.88 GiB, against 10.20 GiB for the same W8A8 build with bf16 embeddings.

Made with w8a8_from_qat.py and w8a8_emb4.py.

Measured on one TPU v5e chip

One v5litepod-1 (15.75 GiB of HBM), vLLM TPU vllm/vllm-tpu@sha256:19a1a052… with the patches below. Suite: a 3,880-record public benchmark read by label probability, paired record for record against google/gemma-4-E4B-it in bf16 (measured on v6e).

Against bf16 it is −0.3 points (95% range −1.0 to +0.5). Throughput is 256 output tokens per request, 3 repeats, on the same chip and image as the W4A16 row; on v5e, int8 x int8 runs natively in the matrix units, and int4 is storage only. GSM8K 94.0% (2,048-token limit), BFCL 91.2%. With 16 prompts of 1,043 tokens it serves 1,231 output tok/s, first token in 0.40 s. Logs and per-record outputs: https://github.com/xbill9/gemma4-dev/tree/main/tpu-vllm-v5e1-4b-w8a8emb4

Serving it

Google Cloud TPU v5e. vLLM's TPU backend has no int8 W8A8 method on its JAX path, no int4 embedding table and no quantized output head, and reads hf_config.text_config; the patches in tpu-vllm-v5e1-12b-w8a8emb4 add them.

GPUs. Untested. vLLM's CUDA path supports compressed-tensors W8A8 linears and int4 embedding groups separately; the E2B build of this scheme serves on a Tesla T4 with vLLM 0.29.

Limitations

• Text only: image and audio inputs are not supported.
• Measured on TPU v5e only.
• Unofficial. Report problems here, not to Google.

License and attribution

Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed format, under the same license. The weights, training and the original model card are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.