Sakura Gemma 4 E2B DualMode — Generation + 768d Embeddings — Q5_K_M GGUF

Research prototype. Community model. Not official from Google.

In 30 seconds

• What: one Gemma 4 E2B-it model that can chat and produce 768-dimensional search vectors for RAG, so you do not need a second embedding model next to it.
• How: the same backbone is used in two modes. Generate is the normal Gemma 4. Embed switches to bidirectional attention with a tiny adapter (+7 MB) and a 768d head.
• One file: Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf holds backbone, adapter, head and metadata.
• Checked: generation is bitwise identical to plain Gemma 4; switching modes leaks no state (36 switches, 0 differences); retrieval keeps 71-79 % of the teacher's MRR on external benchmarks.
• Catch: it needs a small llama.cpp patch (provided); stock llama.cpp cannot load this file.

This is the GGUF version of webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode (Phase 3 / v0.2). EmbeddingGemma 2 (google/embeddinggemma-2) was used only as the teacher during training; it is not in this file and not needed at runtime. The retrieval is EmbeddingGemma-2-distilled, not index-compatible with EmbeddingGemma 2 vectors.

[!IMPORTANT] A small llama.cpp patch is required. Stock llama.cpp cannot load Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf (unmodified llama.cpp b86d2f0: done_getting_tensors: wrong number of tensors; expected 678, got 601; the plain base file loads fine) because the file contains the embedding head and the embed-only LoRA as extra tensors. The patch is in patches/ and applies cleanly to llama.cpp commit f498f864fbc0472004ee1c3616c1188c68eb157f; build instructions are in docs/LLAMA_CPP_RUNTIME.md. For plain generation with stock llama.cpp use the unchanged base file Gemma4-E2B-it-Q5_K_M.gguf from this repo. Stock llama.cpp cannot run the embed mode.

Files

SHA256 Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf: 7c2379bb48b2401a7ef9edd51cd6c53730afd012c36574f72fded4ab3b274848

The dual file is 7.1 MB larger than the base file of the same quantization (head + LoRA, 3,555,328 parameters in F16). Only the backbone is quantized.

Quick start

# build the patched runtime (see docs/LLAMA_CPP_RUNTIME.md)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout f498f864fbc0472004ee1c3616c1188c68eb157f
git apply ../patches/llama.cpp-dualmode.patch
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF && cmake --build build --target llama-sakura-dualmode llama-server -j 8

# one process, one backbone, two contexts (JSON lines on stdin/stdout)
build/bin/llama-sakura-dualmode -m Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf
{"op":"generate","prompt":"<bos><start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n","n":32}
{"op":"embed","text":"task: search result | query: What is the capital of France?"}
{"op":"embed","text":"title: none | text: Paris is the capital and most populous city of France."}

# or: embeddings over HTTP (embed mode only; llama-server --embeddings)
build/bin/llama-server -m Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf --embeddings --ubatch-size 512

Prefixes are part of the text: queries task: search result | query: , documents title: none | text: (or title: <title> | text: ). Truncate to about 128 tokens (training length). llama-server --embeddings serves the embed mode only; the single-process dual mode is llama-sakura-dualmode.

Measured results (Phase 3 adapter, webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode)

All numbers are from this exact file or its F16 sibling, on the same inputs as the Hugging Face model (SciFact test 300 queries / 500 docs, NFCorpus test 249 queries / 500 docs; max 128 tokens). Teacher = EmbeddingGemma 2 on the same texts.

Q5_K_M versus the BF16 model: MRR 98.8 % (SciFact) / 94.8 % (NFCorpus), nDCG@10 98.5 % / 95.5 %. Versus the teacher: 78.1 % / 67.1 % MRR. The remaining distance to the teacher is real (see the HF model card). All embeddings are finite with L2 norm 1.

F16 parity gate (HF vs GGUF): mean cosine 0.9996. For short texts the bf16 Hugging Face reference itself is noisy (a few texts differ from an fp16 run by cos ~0.97); against an fp16 reference the GGUF F16 matches to 1.0000.

Quantization note: Q4_K_M was measured only with the earlier candidate adapter (SciFact 97.7 % / NFCorpus 92.5 % of BF16 MRR vs. 100.2 % / 97.5 % for Q5_K_M), so no Q4 file is shipped.

Generation and state isolation (this file, patched runtime)

Performance (CPU only, 8 threads, AMD Ryzen AI Max+ 392, Windows 11; no GPU backend was built)

No GPU numbers are given because no GPU build was tested.

Known limitations

• Needs the patched runtime for both modes (see above). Only the CPU backend was tested; other backends are unverified.
• No single server process for chat + embeddings yet; llama-sakura-dualmode is the one-process path.
• Embed mode was trained on <=128 tokens, without BOS (the runtime masks the BOS token that llama.cpp always adds).
• Quality: research grade. About 71-79 % of the teacher's MRR on external retrieval; not a drop-in replacement for EmbeddingGemma 2.
• The patch is a private fork-style patch for this release, not an upstream proposal.

Provenance

Backbone google/gemma-4-E2B-it (base commit 3e22461f65e89153144f8adb70e3b8c2cc9845a7), teacher google/embeddinggemma-2 (914f7f89142e33e77833254d9c9b90c3cef7303b), distillation data and recipe: see the HF model card. Gemma terms of use apply. The similarly named webmp3/Sakura-EmbeddingGemma-2-AutoRound-GGUF is a separate project (quantized EmbeddingGemma 2 only).

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura Gemma 4 E2B DualMode — 生成 + 768 维嵌入 — Q5_K_M GGUF

研究原型。社区模型。并非 Google 官方发布。

30 秒了解

• 是什么: 一个 Gemma 4 E2B-it 模型,既能 聊天,又能为 RAG 生成 768 维的搜索向量,因此无需在它旁边再放第二个嵌入模型。
• 如何实现: 同一个主干以两种模式使用。Generate(生成)是正常的 Gemma 4。Embed(嵌入)切换为双向注意力,并加上一个很小的适配器(+7 MB)和一个 768 维的头。
• 一个文件: Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf 包含主干、适配器、头和元数据。
• 已检查: 生成结果与原版 Gemma 4 逐位一致;切换模式不会泄漏状态(36 次切换,0 处差异);在外部基准测试上,检索保留教师模型 MRR 的 71-79 %。
• 注意事项: 需要一个小型 llama.cpp 补丁(已提供);原版 llama.cpp 无法加载此文件。

这是 webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode(Phase 3 / v0.2)的 GGUF 版本。EmbeddingGemma 2(google/embeddinggemma-2)在训练中 仅作为教师模型 使用;它不在此文件中,运行时也不需要。该检索是 EmbeddingGemma-2 蒸馏 的,与 EmbeddingGemma 2 的向量在索引上 不 兼容。

[!IMPORTANT] 需要一个小型 llama.cpp 补丁。 原版 llama.cpp 无法加载 Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf(未修改的 llama.cpp b86d2f0:done_getting_tensors: wrong number of tensors; expected 678, got 601;普通的基础文件可以正常加载),因为该文件额外包含嵌入头和仅用于嵌入的 LoRA 张量。补丁位于 patches/,可干净地应用到 llama.cpp 提交 f498f864fbc0472004ee1c3616c1188c68eb157f 上;构建说明见 docs/LLAMA_CPP_RUNTIME.md。若只用原版 llama.cpp 进行普通生成,请使用本仓库中未改动的基础文件 Gemma4-E2B-it-Q5_K_M.gguf。原版 llama.cpp 无法运行嵌入模式。

文件

Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf 的 SHA256:7c2379bb48b2401a7ef9edd51cd6c53730afd012c36574f72fded4ab3b274848

双模式文件比相同量化的基础文件大 7.1 MB(头 + LoRA,3,555,328 个 F16 参数)。只有主干被量化。

快速开始

# build the patched runtime (see docs/LLAMA_CPP_RUNTIME.md)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout f498f864fbc0472004ee1c3616c1188c68eb157f
git apply ../patches/llama.cpp-dualmode.patch
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF && cmake --build build --target llama-sakura-dualmode llama-server -j 8

# one process, one backbone, two contexts (JSON lines on stdin/stdout)
build/bin/llama-sakura-dualmode -m Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf
{"op":"generate","prompt":"<bos><start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n","n":32}
{"op":"embed","text":"task: search result | query: What is the capital of France?"}
{"op":"embed","text":"title: none | text: Paris is the capital and most populous city of France."}

# or: embeddings over HTTP (embed mode only; llama-server --embeddings)
build/bin/llama-server -m Sakura-Gemma4-E2B-DualMode-Q5_K_M.gguf --embeddings --ubatch-size 512

前缀是文本的一部分:查询 task: search result | query: ,文档 title: none | text: (或 title: <title> | text: )。请截断到约 128 个 token(训练长度)。llama-server --embeddings 只提供嵌入模式;单进程的双模式是 llama-sakura-dualmode。

测量结果(Phase 3 适配器,webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode)

所有数据均来自这个确切的文件或其 F16 同级文件,输入与 Hugging Face 模型相同(SciFact test 300 个查询 / 500 篇文档,NFCorpus test 249 个查询 / 500 篇文档;最多 128 个 token)。教师 = 在相同文本上的 EmbeddingGemma 2。

Q5_K_M 相对于 BF16 模型:MRR 为 98.8 %(SciFact)/ 94.8 %(NFCorpus),nDCG@10 为 98.5 % / 95.5 %。相对于教师:MRR 为 78.1 % / 67.1 %。与教师之间剩余的距离是真实存在的(见 HF 模型卡)。所有嵌入都是有限值,L2 范数为 1。

F16 对照关卡(HF 对 GGUF):平均余弦 0.9996。对于短文本,bf16 的 Hugging Face 参照本身就有噪声(少数文本与 fp16 运行的余弦相差约 0.97);相对于 fp16 参照,GGUF F16 的吻合度为 1.0000。

量化说明: Q4_K_M 只用较早的候选适配器测量过(占 BF16 MRR 的 SciFact 97.7 % / NFCorpus 92.5 %,而 Q5_K_M 为 100.2 % / 97.5 %),因此不发布 Q4 文件。

生成与状态隔离(本文件,打过补丁的运行时)

性能(仅 CPU,8 线程,AMD Ryzen AI Max+ 392,Windows 11;未构建 GPU 后端)

由于没有测试 GPU 构建,因此不提供 GPU 数据。

已知局限

• 两种模式都需要打过补丁的运行时(见上文)。只测试了 CPU 后端;其他后端未经验证。
• 目前还没有同时提供聊天和嵌入的单一服务器进程;llama-sakura-dualmode 是单进程方案。
• 嵌入模式是在 <=128 个 token 上、不带 BOS 训练的(运行时会屏蔽 llama.cpp 总是添加的 BOS token)。
• 质量:研究级。在外部检索上约为教师模型 MRR 的 71-79 %;不能直接替代 EmbeddingGemma 2。
• 该补丁是为本次发布准备的私有分支式补丁,并非提交给上游的提案。

出处

主干 google/gemma-4-E2B-it(基础提交 3e22461f65e89153144f8adb70e3b8c2cc9845a7),教师 google/embeddinggemma-2(914f7f89142e33e77833254d9c9b90c3cef7303b),蒸馏数据和配方:见 HF 模型卡。适用 Gemma 使用条款。名称相近的 webmp3/Sakura-EmbeddingGemma-2-AutoRound-GGUF 是一个独立的项目(仅量化 EmbeddingGemma 2)。