EmbeddingGemma 2, MLX mixed-mxfp8
A quantized MLX build of google/embeddinggemma-2 (740M parameters; text, image, audio and video embeddings in one model.safetensors), made with Krill. Bits: 8 (mxfp8 mixed). Weights file: 1007.4 MB (bf16 original: 1488.9 MB).
All 9 builds, next to Google's original, are in the EmbeddingGemma 2 MLX collection.
What this build is
mxfp8 (8-bit float) on the remaining quantizable tensors (embedding table, MLP, vision and audio weights). The text transformer layers, all self-attention (including vision and audio attention) and the patch embedder are kept dense bf16.
Produced with:
krill quantize <bf16-dir> --mode mxfp8 --dtype bf16 --skip language_model.layers --skip self_attn --skip patch_embedder --output-dir <out>
Gate result: PASS (8-bit class). Drops against bf16: SciFact 0.12, Hindi 0.20, image R@1 0.0 points (negative means the build scored higher). Gate: SciFact and Hindi drop <= 1.0 nDCG point, image R@1 drop <= 1.0 point (variants: mixed-mxfp8, 8bit, 6bit, 6bit-dyn).
Quality against bf16
Cosine similarity is measured against Google's fp32 sentence-transformers reference vectors; retrieval is nDCG@10 or Recall@k. The bf16 column is the unmodified Google weights run through Krill.
| metric | bf16 | this build |
|---|---|---|
| Text cosine to Google fp32 (min / mean) | 1.000 / 1.000 | 0.994 / 0.998 |
| Image cosine (min / mean) | 0.999 / 1.000 | 0.992 / 0.996 |
| Mixed cosine (min / mean) | 1.000 / 1.000 | 0.996 / 0.997 |
| Audio cosine (min / mean) | 1.000 / 1.000 | 0.996 / 0.997 |
| Video cosine (min / mean) | 0.999 / 1.000 | 0.996 / 0.997 |
| SciFact nDCG@10, 768 d | 86.92 | 86.80 |
| SciFact nDCG@10, 256 d | 84.44 | 84.31 |
| Hindi nDCG@10 (IndicQARetrieval) | 72.78 | 72.58 |
| Kannada nDCG@10 (IndicQARetrieval) | 73.25 | 73.18 |
| Flickr image-to-text R@1 / R@5 | 99.5 / 100.0 | 99.5 / 100.0 |
| Flickr text-to-image R@1 / R@5 | 96.1 / 99.7 | 96.1 / 99.6 |
| Clotho audio-to-text R@1 / R@5 | 25.0 / 61.7 | 23.3 / 50.0 |
| Clotho text-to-audio R@1 / R@5 | 30.0 / 58.3 | 26.7 / 58.3 |
| Ranking agreement, Spearman raw / Document | 1.000 / 1.000 | 0.998 / 0.997 |
| Top-1 neighbour unchanged, raw / Document (of 21) | 21 / 21 | 21 / 21 |
SciFact is MTEB mteb/scifact (300 queries, 5,183 documents). Hindi and Kannada are the hi and kn splits of MTEB IndicQARetrieval; the corpora are small, so treat differences under about 0.5 point as noise. Flickr is the first 200 images of the Flickr30k 1K test split. Clotho is 60 clips of the Clotho v2 test split (one clip is 1.7 points of Recall@1). Test sets are small; "PASS" means no measurable loss on these sets, not "equal".
Speed and memory (Apple M4 Pro)
| contender | weights MB | single query p50 ms | batch-32 ~256 tok docs/s | batch-32 ~1k tok docs/s | peak memory MB |
|---|---|---|---|---|---|
| Krill bf16 (fp32 compute) | 1488.9 | 10.2 | 39 | 9 | 3234 |
| Krill mixed-mxfp8 (this build) | 1007.4 | 10.5 | 38 | 8 | 2835 |
| Google sentence-transformers, MPS bf16 | 1488.9 | 26.2 | 26 | 5 | 17283 |
Ollama embeddinggemma-2 (nvfp4, 1.3 GB with towers) |
1300 | 25.2 | 32 | 8 | 1476 |
| Unsloth Q8_0 GGUF (text only, llama.cpp Metal) | 309.9 | 16.4 | 33 | 7 | 1825 |
Unsloth and Ollama are text only; Krill rows include the vision and audio towers in the weights. Image encoding for this build: 388 ms p50 for one 640x480 PNG. Cold first request: 1053 ms. Single-query p95: 14.8 ms.
Method: Apple M4 Pro, 24 GB. One contender at a time, with the server started and stopped for every repetition, 10 discarded warm-up requests, and 3 repetitions per metric reported as the median. Before each repetition the harness required 1-minute load below 3, at least 50% free memory, no thermal limit and AC power. A contender counts as stable when the spread (max - min over the median) of every metric across its 3 repetitions is at most 10%; otherwise it was re-measured. Krill rows use fp32 compute (the default). Full protocol: docs/bench/embeddinggemma2-2026-10-08.md in the Krill repository.
Usage with Krill
krill pull embeddinggemma-2-mxfp8
krill serve
curl localhost:57455/v1/embeddings -H 'Content-Type: application/json' \
-H "Authorization: Bearer $KRILL_API_KEY" -d '{"model":"embeddinggemma-2-mxfp8","input":"What causes the northern lights?","task":"SearchQuery"}'
Use "task": "SearchQuery" for queries and "Document" for documents. Dimensions: 768 (default), 512, 256, 128 (Matryoshka). Context: 8,192 tokens. Compute dtype: float32 (default) or bfloat16; float16 is not supported.
Other builds
mixed-mxfp8, 8bit, 6bit, 6bit-dyn, 5bit, 4bit-g32, 4bit-dyn, 4bit-dyn-text and nvfp4, all under srv-sngh/embeddinggemma-2-mlx-<name>.