🧲 EmbeddingGemma 2, NVFP4 GGUF

Follow me on X @BennyDaBall_OG !

This is an NVFP4 GGUF of google/embeddinggemma-2, made for llama.cpp and LM Studio.

The model is Google DeepMind's work. The architecture, the training, the multimodal embedding space and the task prompts are all theirs. Nothing was trained or tuned here. This repo only changes how the weights are stored, so read Google's model card for what the model can do and how to prompt it.

Yes, it's a 270M embedding model. Yes, it was already small. I made it smaller anyway, and I was careful about it. 🙂

---

🧠 What is this?

• A per-tensor mix, not a blanket 4-bit. Every weight tensor in the text model was checked for how well it holds up in NVFP4. The ones that shrug it off are NVFP4. The ones that don't (the final output projection, the attention value projections, the per-layer input gates and projections, and the FFNs at the very start and end of the network) stay at Q8_0.
• Proper NVFP4. Two-level scaling (an FP32 scale per tensor on top of the FP8 scale per 16 weights), with each block's scale picked for the lowest error instead of the quick default.
• 8-bit activations, set in the file. Blackwell normally runs NVFP4 matmuls with 4-bit activations too. Embeddings are happier with 8-bit, so every NVFP4 tensor is flagged for it. Stock llama.cpp reads the flag; no environment variables needed.
• Full multimodal. The vision and audio encoders ship untouched in BF16, so text, images, video and audio all land in the same 768-dimensional space, just like the original.

🎯 Is it any good?

Yes! It stays right on top of the original:

Retrieval is the average over SciFact, NFCorpus, ArguAna, CosQA (code) and Wikipedia retrieval in German, Bengali and Persian. STS is STS17 across 11 language pairs, and image search is Flickr-1k text-to-image. That's about a third of the size of the original, and the embeddings are closer to BF16 than a standard Q4_K_M at the same size.

One honest note: this model is so small that 4-bit doesn't make it faster. It embeds at roughly 85 to 90 thousand tokens per second on an RTX 5090 either way. You get it in a third of the download and disk space, with quality that's very hard to tell apart from the original.

---

📦 The files

🚀 Run it

llama.cpp (b11658 or newer):

llama-server -m embeddinggemma-2-NVFP4.gguf \
  --mmproj mmproj-embeddinggemma-2-BF16.gguf \
  --embedding --pooling mean -ub 2048 -b 2048 \
  --image-min-tokens 280 --image-max-tokens 280

Then hit /v1/embeddings like any OpenAI-style embeddings endpoint.

• Keep the image flags. EmbeddingGemma 2 sees every image at a 280-token budget. Without those two flags llama.cpp feeds small images at their native size and image search gets noticeably worse. This applies to every GGUF of this model, not just mine.
• -ub has to fit your longest input. The model reads each input in one pass, so raise -ub and -b together if you embed long documents. Leave out --mmproj for text only.
• Use Google's task prompts. For example task: search result | query: in front of queries and title: none | text: in front of documents. The full list is in Google's model card.

LM Studio: set the context length to something like 2048 or 8192 when you load it. The default of 262,144 makes LM Studio fail to load any EmbeddingGemma 2 GGUF, again not just this one. Once loaded it gives exactly the same embeddings as llama.cpp.

Built and tested on an RTX 5090. NVFP4 is a Blackwell format; other GPUs should still run it through llama.cpp's fallback kernels, but I haven't tested that.

---

📜 Credit and license

Original model: google/embeddinggemma-2 by Google DeepMind, released under Apache 2.0, and this quantization keeps the same license. This NVFP4 quantization and release are by BennyDaBall. If it saves you some disk space, a follow on X makes my day. ❤️