Gemma 4 E2B-it QAT, W8A8 int8 linears with int4 embeddings, text only (unofficial repack)
This is an unofficial repack, made and published independently of Google. It is
xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4
with its decoder linears converted from int4 weights with 16-bit math (W4A16) to
int8 weights with int8 math (W8A8). The embedding tables and lm_head are unchanged,
so the two builds differ only in how the decoder linears compute. Google's model card
is kept unchanged in ORIGINAL_README.md.
It exists to measure the INT8 tensor cores on GPUs that have them but no FP8 or bf16,
such as the Tesla T4, not as the default build. On one T4 it decodes at about half the
speed of -emb4, and it is less faithful to the QAT weights. For most uses on a T4,
-emb4 is the better choice.
What it is
| Tensor | Format |
|---|---|
| Decoder linears (attention, MLP, per-layer-embedding projections; 276 modules) | int8 weights, one scale per output channel; activations int8 per token, quantized at run time |
embed_tokens_per_layer, embed_tokens |
int4 group 32, fp16 scales (as in -emb4) |
lm_head |
int4 group 32, untied (as in -emb4) |
| Checkpoint | 3.40 GiB |
No calibration data was used: the activation scales are computed per token at run time.
It is not lossless
The -emb4 build reproduces Google's quantization-aware-trained (QAT) values: they sit on
a 4-bit grid with one step per group of 32 weights, and that build stores the grid. One
int8 scale per row cannot hold every group's step exactly, so each weight moves slightly
(mean relative error 0.89% per linear, worst 1.7%). Rounding the activations to int8 is a
larger change, and QAT never trained for it.
A greedy spot check (8 prompts, up to 160 tokens, temperature 0) against -emb4: 2 of 8
outputs token-identical; the rest reword from token 12 to 44 onward. All eight answers
remain fluent and on-task. One factual slip: asked about HTTP status 418, it cites
RFC 2616 where -emb4 correctly says RFC 2324. This is a spot check, not an evaluation.
Measured on one Tesla T4
vLLM 0.29.0, --dtype float16 --gpu-memory-utilization 0.90 --max-model-len 16384 --max-num-seqs 8. vLLM selects CutlassInt8ScaledMMLinearKernel for the int8 linears
and MarlinLinearKernel for lm_head.
-emb4 (W4A16) |
This build (W8A8) | |
|---|---|---|
| Model loading | 2.86 GiB | 3.62 GiB |
| Decode, one stream, 256 tokens | 109.7 tok/s | 53.8 tok/s |
Decode on a T4 is limited by reading weights from memory, and int8 weights are twice the
bytes of int4, with an extra quantization step per layer on top. Where W8A8 could win is
prefill on long prompts, which is compute-bound; a full concurrency sweep is in the serving
rig, xbill9/gpu-vllm-t4-2b-w4a16.
Related
xbill9/gemma-4-E2B-it-qat-w8a8-int8
uses the same int8 scheme for the linears but keeps the embeddings in bf16 with a tied
bf16 lm_head (6.88 GiB). This build is the one to compare with -emb4, because only the
linears differ.
How it was made
w8a8.py
unpacks each int4 linear of -emb4 to its exact values (level × group step) and re-encodes
it as int8 with one scale per output channel. Modules in other quantization groups
(the embeddings and lm_head) are copied byte for byte.
Serving it
Needs vLLM 0.29 or later (int4 embeddings). On Turing, vLLM's Triton attention also needs a shared-memory clamp; see the serving rig.
vllm serve xbill9/gemma-4-E2B-it-qat-w8a8-ct-text-emb4 --dtype float16 --max-model-len 16384
Restart once after the first start: the first start compiles from scratch and undersizes the KV cache (895,665 tokens on this build's first start).
Limitations
- Text only: image and audio inputs are not supported.
- Tested on one T4 (fp16) with vLLM 0.29.0 only.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.