EmbeddingGemma 2, W8A8 int8 linears with int4 embeddings, text only (unofficial repack)
This is an unofficial repack, made and published independently of Google. It is
google/embeddinggemma-2 with the vision
and audio encoders removed, the text model's linears stored as int8 and the embedding table
as int4, for serving on a CPU. Google's model card is kept unchanged in ORIGINAL_README.md.
The checkpoint is 203.4 MiB, against 1.4 GB for the original (all encoders, bf16).
Its sibling, xbill9/embeddinggemma-2-w4a16-ct-text-emb4
(146.1 MiB), stores the linears as int4.
What it is
| Tensor | Format |
|---|---|
| Text-model linears (attention, MLP, per-layer-embedding gates and projections; 217 modules) | int8 weights, one bf16 scale per output channel; activations declared int8, per tensor, dynamic |
embed_tokens [262144, 512] |
int4 group 32, fp16 scales |
embedding_projection (512 → 768, the last layer) |
bf16, unchanged |
| Norms, scalars | bf16, copied byte for byte |
| Vision and audio encoders | removed (470M of 740M parameters) |
No calibration data was used. Each int8 linear differs from bf16 by 0.75% to 1.1% (relative error of the whole tensor); the int4 embedding table, rounded from bf16 because this model has no QAT export, by 9.0%.
Accuracy
SciFact (BEIR test split, 300 queries, 5,183 documents, max 512 tokens, 768 dimensions,
cosine ranking), on CPU through cpu_load.py:
| build | checkpoint | nDCG@10 | recall@10 | top 10 shared with float32 |
|---|---|---|---|---|
google/embeddinggemma-2, float32 |
1.4 GB | 0.8542 | 0.9437 | 100% |
| This build | 203.4 MiB | 0.8358 (97.9%) | 0.9259 | 85.5% |
| W4A16 sibling | 146.1 MiB | 0.8336 (97.6%) | 0.9342 | 80.7% |
Mean cosine similarity to the float32 original over 16 short texts: 0.993 for documents and 0.995 for queries.
Speed on one vCPU
On Cloud Run, 1 vCPU, one thread, medians of three benchmark runs each, against the original text model in float32: queries take 2.08x less time, and documents (32 per request, about 135 tokens each) embed 1.29x faster. Full method and numbers: embedding-cloudrun.
Using it on CPU
cpu_load.py in this repository loads it with Sentence Transformers. Most linears run as
PyTorch fbgemm dynamic int8 linears (x86). down_proj and per_layer_projection take float
inputs: their int8 weights are dequantized once at load, because a down_proj input has
channels up to about 700x its RMS, which one int8 scale per tensor cannot hold. The int4 table
stays packed in memory and only looked-up rows are unpacked.
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("xbill9/embeddinggemma-2-w8a8-ct-text-emb4")
sys.path.insert(0, path)
from cpu_load import load
model = load(path)
q = model.encode(["What causes the northern lights?"], prompt_name="SearchQuery")
d = model.encode(["Auroras form when charged particles from the Sun hit the upper atmosphere."],
prompt_name="Document")
print(model.similarity(q, d))
Tested with torch 2.14.1 (CPU), transformers 5.19.0 and sentence-transformers 6.1.0.
Limitations
- Text only: image, audio and video inputs are not supported.
- x86 CPUs: the int8 path uses fbgemm.
- Not tested in vLLM or with Transformers' compressed-tensors integration; use
cpu_load.py. - Unofficial. Report problems here, not to Google.
How it was made
repack_text_w8a8_emb4.py
builds it from a snapshot of google/embeddinggemma-2. repack_report.json holds the
per-tensor errors. The SciFact eval is
eval_scifact.py.
License and attribution
EmbeddingGemma 2 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.