EmbeddingGemma 2, MLX 4bit-dyn
A quantized MLX build of google/embeddinggemma-2 (740M parameters; text, image, audio and video embeddings in one model.safetensors), made with Krill. Bits: 4 / 8 attn+ple+proj+last2. Weights file: 472.7 MB (bf16 original: 1488.9 MB).
All 9 builds, next to Google's original, are in the EmbeddingGemma 2 MLX collection.
What this build is
Affine 4-bit, group size 64, with the text projection, text self-attention, the text per-layer-input (ple_block) tensors and the last two text layers kept at 8-bit (group 64). Adds 31 MB over plain 4-bit.
Produced with:
krill quantize <bf16-dir> --bits 4 --group-size 64 --dtype bf16 --protect-bits 8 --protect-group-size 64 --protect 're:^language_model\.embedding_projection$' --protect 're:^language_model\.layers\.\d+\.self_attn\.' --protect 're:^language_model\.layers\.\d+\.ple_block\.' --protect 're:^language_model\.layers\.2[23]\.' --output-dir <out>
Gate result: PASS (4/5-bit class). Drops against bf16: SciFact 0.62, Hindi 0.76, image R@1 0.8 points (negative means the build scored higher). Gate: drops <= 3.0 and text cosine mean >= 0.97.
Quality against bf16
Cosine similarity is measured against Google's fp32 sentence-transformers reference vectors; retrieval is nDCG@10 or Recall@k. The bf16 column is the unmodified Google weights run through Krill.
| metric | bf16 | this build |
|---|---|---|
| Text cosine to Google fp32 (min / mean) | 1.000 / 1.000 | 0.983 / 0.993 |
| Image cosine (min / mean) | 0.999 / 1.000 | 0.981 / 0.992 |
| Mixed cosine (min / mean) | 1.000 / 1.000 | 0.986 / 0.992 |
| Audio cosine (min / mean) | 1.000 / 1.000 | 0.992 / 0.994 |
| Video cosine (min / mean) | 0.999 / 1.000 | 0.989 / 0.991 |
| SciFact nDCG@10, 768 d | 86.92 | 86.30 |
| SciFact nDCG@10, 256 d | 84.44 | 82.94 |
| Hindi nDCG@10 (IndicQARetrieval) | 72.78 | 72.03 |
| Kannada nDCG@10 (IndicQARetrieval) | 73.25 | 73.00 |
| Flickr image-to-text R@1 / R@5 | 99.5 / 100.0 | 99.5 / 100.0 |
| Flickr text-to-image R@1 / R@5 | 96.1 / 99.7 | 95.3 / 99.5 |
| Clotho audio-to-text R@1 / R@5 | 25.0 / 61.7 | 23.3 / 65.0 |
| Clotho text-to-audio R@1 / R@5 | 30.0 / 58.3 | 28.3 / 61.7 |
| Ranking agreement, Spearman raw / Document | 1.000 / 1.000 | 0.992 / 0.986 |
| Top-1 neighbour unchanged, raw / Document (of 21) | 21 / 21 | 19 / 19 |
SciFact is MTEB mteb/scifact (300 queries, 5,183 documents). Hindi and Kannada are the hi and kn splits of MTEB IndicQARetrieval; the corpora are small, so treat differences under about 0.5 point as noise. Flickr is the first 200 images of the Flickr30k 1K test split. Clotho is 60 clips of the Clotho v2 test split (one clip is 1.7 points of Recall@1). Test sets are small; "PASS" means no measurable loss on these sets, not "equal".
Speed and memory (Apple M4 Pro)
| contender | weights MB | single query p50 ms | batch-32 ~256 tok docs/s | batch-32 ~1k tok docs/s | peak memory MB |
|---|---|---|---|---|---|
| Krill bf16 (fp32 compute) | 1488.9 | 10.2 | 39 | 9 | 3234 |
| Krill 4bit-dyn (this build) | 472.7 | 9.0 | 34 | 7 | 2398 |
| Google sentence-transformers, MPS bf16 | 1488.9 | 26.2 | 26 | 5 | 17283 |
Ollama embeddinggemma-2 (nvfp4, 1.3 GB with towers) |
1300 | 25.2 | 32 | 8 | 1476 |
| Unsloth Q8_0 GGUF (text only, llama.cpp Metal) | 309.9 | 16.4 | 33 | 7 | 1825 |
Unsloth and Ollama are text only; Krill rows include the vision and audio towers in the weights. Image encoding for this build: 403 ms p50 for one 640x480 PNG. Cold first request: 992 ms. Single-query p95: 13.0 ms.
Method: Apple M4 Pro, 24 GB. One contender at a time, with the server started and stopped for every repetition, 10 discarded warm-up requests, and 3 repetitions per metric reported as the median. Before each repetition the harness required 1-minute load below 3, at least 50% free memory, no thermal limit and AC power. A contender counts as stable when the spread (max - min over the median) of every metric across its 3 repetitions is at most 10%; otherwise it was re-measured. Krill rows use fp32 compute (the default). Full protocol: docs/bench/embeddinggemma2-2026-10-08.md in the Krill repository.
Usage with Krill
krill pull embeddinggemma-2-4bit-dyn
krill serve
curl localhost:57455/v1/embeddings -H 'Content-Type: application/json' \
-H "Authorization: Bearer $KRILL_API_KEY" -d '{"model":"embeddinggemma-2-4bit-dyn","input":"What causes the northern lights?","task":"SearchQuery"}'
Use "task": "SearchQuery" for queries and "Document" for documents. Dimensions: 768 (default), 512, 256, 128 (Matryoshka). Context: 8,192 tokens. Compute dtype: float32 (default) or bfloat16; float16 is not supported.
Other builds
mixed-mxfp8, 8bit, 6bit, 6bit-dyn, 5bit, 4bit-g32, 4bit-dyn, 4bit-dyn-text and nvfp4, all under srv-sngh/embeddinggemma-2-mlx-<name>.