EmbeddingGemma 2 int4: multimodal text, image and audio embedding model in 486 MiB

This is google/embeddinggemma-2 with all three towers packed to mostly 4 bits. It keeps the same API and the same 768-dimension embeddings. It is 66% smaller on disk than the bf16 original and loses under 1% retrieval quality in the tests below. Nothing was trained. The weights are rounded once, after training, with no calibration data. The few layers that were most sensitive to rounding stay at 8 bits.

At a glance

The memory saving is far smaller than the disk saving, because Python, torch and the activations stay the same size. If you care about download size or storage, this helps a lot. If you care about process RAM, expect roughly 14 to 22% lower peak memory, not 66%.

This is the second release. The first release (451.5 MiB, everything in 4 bits) is revision 798b305f3f1eb292c35543ceaead09821d7dd233 if you need the smaller file: snapshot_download(repo, revision="798b305f3f1eb292c35543ceaead09821d7dd233"). The second release adds 35 MiB of 8-bit layers and changes the top result less often, see "Limits". The memory and GPU speed numbers were measured on the first release. They have not been repeated on this one, which holds 35 MiB more weights.

Quick start

You need transformers==5.19.0, sentence-transformers>=6.1 and accelerate.

import sys, torch
from huggingface_hub import snapshot_download

path = snapshot_download("ewin-reg/embeddinggemma-2-int4-text-image-audio-embedding")
sys.path.insert(0, path)
import qpack3

st = qpack3.load_st(path, dtype=torch.bfloat16, device="cuda")  # or "cpu"
q = st.encode(["task: search result | query: a red bicycle"])
d = st.encode(["title: none | text: a bicycle leaning on a wall"])
img = st.encode([pil_image])            # list of PIL images
aud = st.encode({"audio": waveform})    # 16 kHz mono float32 array

Use the original model's prompts. Queries start with task: search result | query: and documents with title: none | text: . The loader builds the model on empty weights and attaches the packed tensors, so it never allocates a full-precision copy.

Making it faster on a GPU

Wrap encoding in fp16 autocast. This needs no change to the weights. I measured it on the first release, on a T4 with 1,000 COCO images and the model loaded in fp32:

st = qpack3.load_st(path, dtype=torch.float32, device="cuda")
with torch.autocast("cuda", dtype=torch.float16):
    img = st.encode(pil_images, batch_size=8)

The 140 soft token mode (st[0].processor.image_processor.max_soft_tokens = 140) halves the patches per image. It is the fastest setting, but it costs about 0.7% NDCG@10 for text-to-image, 1.9% Recall@1 for text-to-image, and changes the top-1 image for 27% of image-to-text queries and 9% of text-to-image queries. Use it only if you do not need exact top-1 ranking. The default of 280 is what the quality numbers below describe. I did not test fp16 with a model loaded in bf16.

Limits to know before you use it

• Keep max_soft_tokens at 280 or lower, which is the default. The vision position table was cut to 2,520 rows. That is exact for 280 soft tokens and wrong above it.
• Rounding changes which result ranks first for some queries, even though NDCG@10 and Recall@10 stay within 0.25%. Against the original, with the original's text embeddings on the other side, the top image changes for 4.1% of image-to-text queries and 2.6% of text-to-image queries (the first release: 4.8% and 4.4%; the text-to-image drop is statistically significant, McNemar p = 0.0005). Counting the exact top caption instead of the top image, the image-to-text figure is 8.2%. With both sides packed it is 15.3% for image-to-text and 3.1% for text-to-image. If you need the exact top-1 of the original model, do not use this.
• Audio was checked on easy data only, so treat audio as less verified than text and vision.
• Quality was measured in fp32. The memory and speed runs used bf16 on CPU.

Retrieval quality

Each row compares this release with the unquantized original on the same inputs, loading the packed files through this repository's loader. Compute was fp32 on a T4 GPU.

For the first row, the 95% bootstrap upper bounds on the loss are 0.60% for NDCG@10 and 0.46% for Recall@10. On the 1,000-image caption subset the Recall@10 loss was 0.66% (upper bound 1.37%). On all 5,000 images the loss is 0.23%, with a cluster bootstrap over images (2,000 resamples) that puts the upper bound at 0.41% for NDCG@10 and Recall@10. Both upper bounds are under 1%. The 1,000-image figure was a noisier estimate of the same model, not a different result. Both audio scores are 100%, so that test shows no loss but is too easy to tell the models apart. The mean cosine similarity between the image embeddings and the original's is 0.9959, and between the audio embeddings and the original's about 0.98.

Memory and speed

I measured these on the first release (451.5 MiB), on a Kaggle CPU kernel with bf16 compute and transformers 5.19.0. Each run used a fresh process and loaded the repository from a clean download. The figure is peak resident memory (VmHWM), and the original is google/embeddinggemma-2 loaded the same way, audio tower included.

The two runs were on different Kaggle machines, and the original's own peak differed by 8 to 11% between them, so read the saving as a range. Forward time matched the original in runs 1 and 2. In run 3 the second release took 12% to 20% longer on the image workload, which I did not repeat, so I cannot say whether that is noise or the 8-bit layers. Resident memory right after load does not compare fairly, because the original's weights are memory-mapped and load lazily, so peak is the fair number. Memory-mapping the packed files did not help: mapping everything saved about 3% more and slowed the forward pass by 26% to 65%, and mapping only the audio tower changed peak memory by 1% to 2%.

What is in the 486 MiB

How it was packed

• Int4 covers most linear layers in the text tower, the text token table, the vision block linears and the audio block linears. Groups are 16 input columns wide, with asymmetric min-max rounding.
• Int8, in the same groups of 16, covers the most rounding-sensitive layers: the text tower's non-layer linears (including the output projection) and text layers 10, 11, 14, 22 and 23, found with a data-free sweep that quantizes one layer at a time and measures the cosine change of caption embeddings, and vision blocks 12 to 15, from the same kind of sweep on image embeddings. The text tower's output projection and layer 23 were 5 to 15 times more sensitive than any other text layer. The extra cost is 35 MiB.
• Each group's scale and minimum are stored as 8-bit integers per row, with fp16 row bounds.
• The vision patch projection, the two projectors, the convolutions and all normalization weights stay in bf16.
• The vision position table is cut from 10,240 to 2,520 rows per axis. At 280 soft tokens this is lossless, and I checked the embeddings are bit-identical.
• I compared real byte packing with the simulated rounding used in the group-size sweep. The maximum absolute difference is 0 on the tested layers.

I picked group size 16 after comparing 16, 32, 64 and 128 on the same images. Group 16 had the smallest quality loss and the lowest top-1 change rate for image to text in a vision-tower-only simulation, 7.5%, against 10.2% at group 32 and 10.9% at group 128. 3-bit failed the 1% bar. I also tried clipping each group's range to minimize squared rounding error. It lowered weight error but made retrieval worse (image-and-text worst loss 0.49% against 0.22%, Recall@1 loss 1.32% against 0.32%, caption Recall@10 loss 1.20% against 0.66%) and left the top-1 changes where they were (17.1% against 17.0% for image to text with both sides packed), so I did not use it.

Not measured

• Any dataset other than COCO captions and LibriSpeech.
• GPU memory and speed on the second release (the GPU numbers were measured on the first release).
• Retrieval quality in bf16.
• Memory at longer text lengths, or on a GPU.
• Video inputs.

License and credit

Apache-2.0, like the original. The weights come from Google's google/embeddinggemma-2, and this repository only repacks them.

Inference optimization and efficiency techniques

What this package does to cut memory and time, what I measured, and what I tried and dropped. Numbers come from the sections above.

Used and measured

Tried and dropped

• Data-free clipping of group ranges. It lost more quality than plain min-max rounding.
• Memory-mapping the packed files. It saved about 3% memory and slowed the forward pass by 26% to 65%.
• Group sizes of 32, 64 and 128. They changed more top-1 results than group 16.
• 3-bit weights. They failed the 1% bar.

Not tried, so no claim

• A fused dequantize-and-multiply kernel, or tiled dequantization. The loader expands each packed linear to bf16 before the matmul, so latency gains on CPU would need a kernel I have not written.
• torch.compile, ONNX, or other runtime export.
• Batching or caching strategies beyond what sentence-transformers already does.

The limit on CPU memory is structural. Python, torch and the activations stay the same size whatever the weight format, so peak process memory fell by 14% to 24%, not by the 66% the file shrank.