EmbeddingGemma 2, W4A16 int4 linears with int4 embeddings, text only (unofficial repack)

This is an unofficial repack, made and published independently of Google. It is google/embeddinggemma-2 with the vision and audio encoders removed and the text model's weights stored as int4. Google's model card is kept unchanged in ORIGINAL_README.md.

The checkpoint is 146.1 MiB, against 1.4 GB for the original (all encoders, bf16). Its sibling, xbill9/embeddinggemma-2-w8a8-ct-text-emb4 (203.4 MiB), keeps the linears at int8 and runs them with int8 math on CPU.

What it is

The layout is compressed-tensors pack-quantized (weight_packed int32, weight_scale, weight_shape), the same as the Gemma 4 -w4a16-ct builds. No calibration data was used.

Accuracy

EmbeddingGemma 2 has no quantization-aware-trained (QAT) export, so every group of 32 weights is rounded from bf16: the step is the candidate amax / m, m in 7.0 to 8.0, with the least squared error after rounding the step to bf16. Each int4 linear differs from bf16 by 8.5% to 10.5% (relative error of the whole tensor); the embedding table by 9.0%.

SciFact (BEIR test split, 300 queries, 5,183 documents, max 512 tokens, 768 dimensions, cosine ranking), all on CPU in float32 activations:

Mean cosine similarity to the float32 original over 16 short texts: 0.984 for documents and 0.986 for queries (W8A8 sibling: 0.993 and 0.995).

The format was checked with compressed-tensors 0.19.0 itself: its PackedQuantizationCompressor decodes all 218 packed tensors to exactly the values cpu_load.py uses.

Using it on CPU

cpu_load.py in this repository loads it with Sentence Transformers, without compressed-tensors. It dequantizes the int4 linears to float32 once at load, so the saving is in download and image size; the int4 table stays packed in memory and only looked-up rows are unpacked.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("xbill9/embeddinggemma-2-w4a16-ct-text-emb4")
sys.path.insert(0, path)
from cpu_load import load

model = load(path)
q = model.encode(["What causes the northern lights?"], prompt_name="SearchQuery")
d = model.encode(["Auroras form when charged particles from the Sun hit the upper atmosphere."],
                 prompt_name="Document")
print(model.similarity(q, d))

Tested with torch 2.14.1 (CPU), transformers 5.19.0 and sentence-transformers 6.1.0.

Speed on one vCPU

On Cloud Run, 1 vCPU, one thread, three runs each alternating with the W8A8 sibling on the same day (medians, range in brackets):

Documents are 32 per request, about 135 tokens each. This build is slower because its linears run in float32, and it needs more memory: loading plus one 32-document batch peaks at 2,127 MiB (W8A8: 1,856 MiB). Choose it for the smaller download; choose W8A8 for CPU serving. Full method: embedding-cloudrun.

Limitations

• Text only: image, audio and video inputs are not supported.
• Not tested in vLLM. Transformers' from_pretrained with its compressed-tensors integration fails on this checkpoint at the quantized embedding table; use cpu_load.py.
• Unofficial. Report problems here, not to Google.

How it was made

repack_text_w4a16_emb4.py builds it from a snapshot of google/embeddinggemma-2; its verify command rereads both checkpoints and checks every tensor. repack_report.json holds the per-tensor errors. The SciFact eval is eval_scifact.py.

License and attribution

EmbeddingGemma 2 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.