Sakura EmbeddingGemma 2 HighQuality GGUF

Five GGUF quantizations of google/embeddinggemma-2 (768-d, multilingual) at 177 to 244 MB. Each tensor group gets its own precision, and the importance matrix comes from our own calibration text. Community build, not from Google.

Pick a file

Distance is measured against the BF16 model: 1 - cosine over 1,400 held-out query and passage pairs (lower is better), and how often the nearest neighbour is the same as with BF16 (top-1). Sizes are bytes on disk.

Q4 is the smallest file that is still useful. Q5 is the sweet spot. Q6 is close to lossless for retrieval; at 245 MB it is level with the best published files except AtomicChat's.

Where this is ahead and where it is not

We measured with the same set and the same BF16 reference as the published numbers of the other builds, so the rows below are comparable. The other files are listed as they were published; we did not re-measure them.

In short: this series is clearly ahead in the Q4 class on distance, about level in Q5, and level with AutoRound and ahead of Unsloth in Q6, but behind AtomicChat's AD-Q6_K at the same size. The 1,400 pairs are one test set; differences of a point in top-1 are within the noise you should expect.

How it was built

• Source: google/embeddinggemma-2 revision 914f7f89142e33e77833254d9c9b90c3cef7303b, converted to BF16 GGUF with llama.cpp's convert_hf_to_gguf.py.
• Importance matrix: computed on the BF16 file with llama-imatrix over 16,460 query/passage units (4.47 M characters) from 13 public retrieval sets in the EmbeddingGemma prompt format (task: search result | query: ..., title: none | text: ...): Natural Questions, GooAQ, SQuAD, TriviaQA, MS MARCO, SPECTER, S2ORC, CodeSearchNet, StackExchange duplicates, Yahoo Answers, ELI5, GermanQuAD, WikiMatrix. None of the benchmark passages are in it (0 of 1,400 evaluation passages share an 80-character span with the calibration text). The matrix file is included; it holds activation statistics only, no text.
• Quantization: llama-quantize --imatrix with per-tensor overrides:

"Blocks" are the attention and feed-forward weights of those layers. per_layer_model_proj is converted to Q8_0 in a second step because llama-quantize leaves it unquantized.

• Measurement: every file is run through llama-server --embeddings and compared with the BF16 vectors, using the protocol, evaluation set and BF16 reference published by AtomicChat in AtomicChat/embeddinggemma-2-GGUF-metrics. Only that measurement protocol is shared; the files here use our own calibration data and our own importance matrix.
• Tools: llama-imatrix and llama-quantize come from our llama.cpp working tree (it carries an unrelated patch for another project that does not touch quantization).
• Runtime: all five files were measured with unmodified llama.cpp (commit b86d2f0, CPU build). No patch is needed.

Use

llama-server -m embeddinggemma-2-Sakura-HQ-Q5.gguf --embeddings --pooling mean -c 2048

EmbeddingGemma expects task prefixes:

• query: task: search result | query: <text>
• document: title: none | text: <text>

The model returns 768 numbers. For smaller vectors keep the first 512, 256 or 128 and normalize again (Matryoshka).

Files and checks

SHA256SUMS lists every file. The five files were measured in exactly the form they are uploaded here; the only edit after quantization is the general.name and general.finetune strings, and the tensor data was checked to be byte-identical before and after.

License and credits

The weights are derived from google/embeddinggemma-2, which is published under Apache-2.0; this repository uses the same license. Model by Google. Quantization, calibration and measurements by webmp3. The measurement protocol is by AtomicChat. This is a community build and not endorsed by Google or AtomicChat.

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura EmbeddingGemma 2 HighQuality GGUF

google/embeddinggemma-2(768 维,多语言)的五个 GGUF 量化版本,文件大小 177 至 244 MB。每个张量组使用各自的精度,重要性矩阵(importance matrix)来自我们自己的校准文本。这是社区构建版本,并非 Google 官方发布。

如何选择文件

距离是相对于 BF16 模型测得的:在 1,400 对留出的查询与段落上计算 1 - cosine(越低越好),以及最近邻与 BF16 结果相同的比例(top-1)。大小为磁盘上的字节数。

Q4 是仍然实用的最小文件。Q5 是最佳平衡点。Q6 在检索任务上接近无损;在 245 MB 的大小下,它与已发布的最佳文件基本持平,AtomicChat 的文件除外。

领先之处与不足之处

我们使用与其他构建版本已发布数据相同的测试集和相同的 BF16 参考进行测量,因此下表中的各行是可比较的。其他文件按其发布时的数据列出,我们没有重新测量。

简而言之:这一系列在 Q4 级别的距离指标上明显领先,在 Q5 级别大致持平,在 Q6 级别与 AutoRound 持平且领先于 Unsloth,但在相同大小下落后于 AtomicChat 的 AD-Q6_K。这 1,400 对数据只是一个测试集;top-1 上相差一个百分点以内属于预期的噪声范围。

构建方式

• 来源: google/embeddinggemma-2,修订版本 914f7f89142e33e77833254d9c9b90c3cef7303b,使用 llama.cpp 的 convert_hf_to_gguf.py 转换为 BF16 GGUF。
• 重要性矩阵: 在 BF16 文件上使用 llama-imatrix 计算,数据为来自 13 个公开检索数据集的 16,460 个查询/段落单元(4.47 M 字符),采用 EmbeddingGemma 提示格式(task: search result | query: ...、title: none | text: ...):Natural Questions、GooAQ、SQuAD、TriviaQA、MS MARCO、SPECTER、S2ORC、CodeSearchNet、StackExchange duplicates、Yahoo Answers、ELI5、GermanQuAD、WikiMatrix。其中不含任何基准测试段落(1,400 个评测段落中,0 个与校准文本有 80 字符的重叠片段)。矩阵文件已一并提供;它只包含激活统计信息,不含任何文本。
• 量化: 使用 llama-quantize --imatrix,并对各张量单独指定类型:

“块”指这些层的注意力权重和前馈权重。per_layer_model_proj 在第二步中转换为 Q8_0,因为 llama-quantize 会让它保持未量化状态。

• 测量: 每个文件都通过 llama-server --embeddings 运行,并与 BF16 向量比较,所用的协议、评测集和 BF16 参考均由 AtomicChat 发布于 AtomicChat/embeddinggemma-2-GGUF-metrics。共享的只有该测量协议;本仓库的文件使用我们自己的校准数据和我们自己的重要性矩阵。
• 工具: llama-imatrix 和 llama-quantize 来自我们自己的 llama.cpp 工作树(其中带有一个与量化无关的、服务于另一个项目的补丁)。
• 运行时: 五个文件均使用未经修改的 llama.cpp(提交 b86d2f0,CPU 构建)测量。无需任何补丁。

使用

llama-server -m embeddinggemma-2-Sakura-HQ-Q5.gguf --embeddings --pooling mean -c 2048

EmbeddingGemma 需要任务前缀:

• 查询:task: search result | query: <text>
• 文档:title: none | text: <text>

模型返回 768 个数值。若需要更小的向量,保留前 512、256 或 128 个数值并重新归一化即可(Matryoshka)。

文件与校验

SHA256SUMS 列出了每个文件。这五个文件是以与上传时完全相同的形式测量的;量化之后唯一的改动是 general.name 和 general.finetune 字符串,并且已核对张量数据在改动前后逐字节一致。

许可证与致谢

权重源自 google/embeddinggemma-2,该模型以 Apache-2.0 发布;本仓库使用相同的许可证。模型由 Google 提供。量化、校准和测量由 webmp3 完成。测量协议来自 AtomicChat。这是社区构建版本,未获得 Google 或 AtomicChat 的认可。