EmbeddingGemma 2 — GGUF, lighter (Q6_K text, split encoders)

EmbeddingGemma 2 by Google DeepMind for llama.cpp, packed to download less:

• the text model quantized to Q6_K (246 MB instead of Q8_0's 310 MB), with the same search quality in our tests;
• the image and audio encoders split into two files, so an app that only needs pictures doesn't download the audio encoder (226 MB for images, 328 MB for audio, instead of 555 MB for both).

Made for Pastemind, a clipboard history and launcher for macOS that searches by meaning. Nothing was retrained or fine-tuned.

Files

Use the text model with either encoder, or both (each as its own mmproj, in its own mtmd context).

How they were made

• Text model: llama-quantize … Q6_K (llama.cpp b11461) from embeddinggemma-2-BF16.gguf of ggml-org/embeddinggemma-2-GGUF at revision bfcd298762cc34d0357ece5ebdd31791a3a374d8 (converted from google/embeddinggemma-2). Never F16: the model's activations overflow float16.
• Encoders: mmproj-embeddinggemma-2-Q8_0.gguf from the same revision, split with llama.cpp's gguf-py: the vision file keeps the v.* tensors and mm.input_projection, the audio file the a.* tensors and mm.a.input_projection, each with the other encoder's metadata removed and its clip.has_*_encoder flag set to false. Tensors are copied byte for byte: vectors from the split files equal those from the original file (cosine 1.000000 on 10 pictures, a video and 3 recordings).

Quality against Q8_0

Pastemind's evaluations (llama.cpp, Metal, M1 Pro; pictures at 140 soft tokens):

The queries are in several languages; numbers are top-1 hits. Q6_K's vectors are close enough to Q8_0's that an index made with one can be searched with the other. Speed is the same (a query ~9 ms, a picture ~180 ms).

Use

llama-embedding -m embeddinggemma-2-Q6_K.gguf --pooling mean -p "task: search result | query: a dog on the beach"

Images and audio go through mtmd (mtmd_init_from_file with the vision or the audio file), their output rows then take their place among the text model's tokens, mean-pooled into one 768-d vector in the same space as text. Prompts and sizing are those of the original model: see google/embeddinggemma-2.

License

Apache License 2.0, as the original model (see LICENSE). EmbeddingGemma 2 is by Google DeepMind; the GGUF conversion is by ggml-org. These files change the format only, as described above. Not affiliated with or endorsed by Google.