Sakura MicroQuality — Kolibri-1 365 experts (GGUF, three sizes)

Made in Germany: Kolibri-1 is the open 78B model of the German company Aleph Alpha, a mixture-of-experts model (3.5B active parameters, 384 routed experts plus one shared expert per layer) for German and English, with a thinking mode and tool calling. This repository holds the same files as Sakura-MicroQuality-Kolibri-1-GGUF with the 19 least used routed experts of every layer removed (365 of 384), about 4.5 % smaller. This repository holds three GGUF files of different size made from the Q8_0 GGUF of the original model (near-lossless; all quality numbers use it as the reference) with our measured mixed-codec method. Pick one on the Files page; the table below says what each file is. Independent community quantization, not an official Aleph Alpha release. Part of the Sakura Micro line.

Sakura MicroQuality

Smaller than the standard quant of the same class, and much closer to the original. Every file here is measured against the standard llama.cpp quant of its size class (same importance matrix, same texts, same machine) and only files that beat it are published. Measured as KL divergence to the Q8_0 source on 4 held-out texts (en, dev, wiki, de); lower is better.

Needs a patched llama.cpp for now. The kolibri1 architecture is not in mainline llama.cpp yet (feature request #29922). Apply the patch of Hob-forge/Kolibri-1-GGUF to llama.cpp commit 836d571, the same as for the other community GGUF files of this model. This card will be updated when upstream support lands.

Quality at a glance

Same method, same texts, same machine for every file; lower KL divergence means closer to the near-lossless Q8_0 reference:

• Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf vs llama.cpp default IQ2_XS (21.41 GiB): KL divergence to the near-lossless Q8_0 reference 58 % lower on average over all 4 texts (1.41 GiB smaller).
• Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf vs llama.cpp default IQ3_XXS (28.12 GiB): KL divergence to the near-lossless Q8_0 reference 50 % lower on average over all 4 texts (0.67 GiB smaller).
• Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf vs llama.cpp default IQ4_XS (38.91 GiB): KL divergence to the near-lossless Q8_0 reference 39 % lower on average over all 4 texts (2.60 GiB smaller).

Why: the bit budget per tensor is allocated from measured sensitivity (see How it was made), not from fixed rules. Full table below.

Which file should I take?

The files

"Tensor types by size" lists the share of the file's bytes per ggml type (all are standard llama.cpp types, so the files run in stock llama.cpp). Quality columns: mean KL divergence of the quantized model's next-token distribution against the Q8_0 source on held-out text (lower is better) and how often the most likely token is unchanged (higher is better).

Comparison at similar size

Same texts, same context, same chunks, measured by us with llama-perplexity (near-lossless Q8_0 source = reference; PPL of the Q8_0 reference: en: 2.460, dev: 1.928, wiki: 24.848, de: 2.908). The reference quants are: llama.cpp default recipe, same importance matrix (ours), measured the same way. Sorted by size.

Method: 12 chunks of 512 tokens per text; texts: English held-out text, developer-text held-out set, English encyclopedia held-out text (not used in any calibration), German held-out text (not used in any calibration). Single measurement on one machine; small KLD differences at similar size are not a quality ranking.

How it was made

1. The source is the Q8_0 GGUF from Hob-forge/Kolibri-1-GGUF (converted from Aleph Alpha's FP8 checkpoint; near-lossless). We did not convert from the 156 GB BF16 weights, so the reference for every number on this card is that Q8_0. The importance matrix was computed by us (llama-imatrix) on German and English text, code and the model's own chat answers with thinking.
2. Every large weight matrix was quantized once per candidate type (Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, IQ4_XS, Q4_K, Q8_0) with llama-quantize and the importance matrix; the error of each choice was estimated per matrix (importance-weighted) and scaled by a sensitivity factor measured with real KL divergence: every group of tensors (for example the FFN down projections, the attention value projections, the first and last layers) was moved alone to a lower-bit type, and the KL divergence it caused was compared with its quantization error.
3. An exact budget allocation (multiple-choice knapsack over bytes) picked one type per matrix for each target size; the final model was assembled from the stored tensors without re-quantizing. We do not publish per-tensor choices. Norms and small tensors stay at high precision.

Limits

• Measured only with KL divergence and perplexity on short held-out texts and not with downstream benchmarks; do not read it as a task-quality claim.
• Quantization always costs some quality; the smallest file costs the most.
• Requires a patched llama.cpp until upstream support lands (see the banner at the top). All numbers on this card were measured with that build (Vulkan backend).
• What was removed: in every layer the 19 routed experts that our calibration texts (German and English text, code, and the model's own chat answers with thinking) selected least often. On those texts only about 0.07 % of all expert selections fell on the removed experts (at most 0.23 % in any layer); 375 of the 950 removed experts were never selected at all. The cut slices the already quantized expert tensors, router weights and router bias, with no requantization and no retraining; top-6 routing is unchanged. The files are byte-for-byte slices of the matching files in Sakura-MicroQuality-Kolibri-1-GGUF. Content that our texts do not cover may rely on removed experts, so check your own use case.
• Requantizing from a Q8_0 source adds a very small extra error compared with converting from BF16; the quality columns compare every file against that same Q8_0 source of the full model.
• Recommended settings from the model card: temperature 1.0, top-p 0.97, top-k 128; reasoning effort is set per request (none, low, medium, high).

Credits

• Original model: Kolibri-1 (Aleph Alpha, Apache-2.0), Apache-2.0 license (included as LICENSE).
• Q8_0 source and the llama.cpp patch: Hob-forge/Kolibri-1-GGUF (Hob-forge / Seraphiel102). Reference quants: llama-quantize default recipes of the full model with our importance matrix.
• GGUF format and tools: ggml-org/llama.cpp (MIT).

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura MicroQuality — Kolibri-1 365 位专家(GGUF,三种大小)

德国制造: Kolibri-1 是德国公司 Aleph Alpha 的开源 78B 模型,为混合专家(MoE)模型(3.5B 激活参数,每层 384 个路由专家加 1 个共享专家),面向德语和英语,带有思考模式和工具调用。本仓库包含与 Sakura-MicroQuality-Kolibri-1-GGUF 相同的文件,但每层移除了使用最少的 19 个路由专家(384 个中保留 365 个),大小约小 4.5 %。本仓库包含 三个不同大小的 GGUF 文件,它们由原模型的 Q8_0 GGUF(近乎无损;所有质量数据均以它为参照)通过我们实测得出的混合编码方法制作。请在 Files 页面选择其中一个;下表说明了每个文件的用途。 这是独立的社区量化版本,不是 Aleph Alpha 的官方发布。属于 Sakura Micro 系列。

Sakura MicroQuality

比同级别的标准量化更小,并且更接近原始模型。 这里的每个文件都与其大小级别的标准 llama.cpp 量化(相同的重要性矩阵、相同的文本、同一台机器)进行了对比测量,只有优于它的文件才会发布。测量方式为在 4 个留出文本(en、dev、wiki、de)上相对于 Q8_0 源文件的 KL 散度;越低越好。

目前需要打过补丁的 llama.cpp。 kolibri1 架构尚未进入 llama.cpp 主线(功能请求 #29922)。请将 Hob-forge/Kolibri-1-GGUF 的补丁应用到 llama.cpp 提交 836d571 上,与该模型的其他社区 GGUF 文件相同。上游支持落地后,本卡片会更新。

质量概览

每个文件使用相同的方法、相同的文本、同一台机器;KL 散度越低,越接近近乎无损的 Q8_0 参照:

• Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf 对比 llama.cpp 默认的 IQ2_XS (21.41 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 58 %(小 1.41 GiB)。
• Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf 对比 llama.cpp 默认的 IQ3_XXS (28.12 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 50 %(小 0.67 GiB)。
• Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf 对比 llama.cpp 默认的 IQ4_XS (38.91 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 39 %(小 2.60 GiB)。

原因:每个张量的比特预算是根据实测的敏感度分配的(见 制作方式),而不是按固定规则。完整表格见下文。

我该选哪个文件?

文件

“各张量类型(按大小)”列出每个 ggml 类型占文件字节数的比例(全部是标准的 llama.cpp 类型,因此这些文件可在原版 llama.cpp 中运行)。质量列:量化模型的下一个 token 分布相对于 Q8_0 源文件在留出文本上的平均 KL 散度(越低越好),以及最可能的 token 保持不变的频率(越高越好)。

相近大小的比较

相同的文本、相同的上下文、相同的分块,由我们使用 llama-perplexity 测量(近乎无损的 Q8_0 源文件 = 参照;Q8_0 参照的 PPL:en: 2.460,dev: 1.928,wiki: 24.848,de: 2.908)。参照量化为:llama.cpp 默认配方、相同的重要性矩阵(我们的)、以相同方式测量。按大小排序。

方法:每个文本取 12 个块,每块 512 个 token;文本为:英文留出文本、开发者文本留出集、英文百科留出文本(未用于任何校准)、德文留出文本(未用于任何校准)。这是在一台机器上的单次测量;相近大小下 KLD 的细小差异并不构成质量排名。

制作方式

1. 来源是 Hob-forge/Kolibri-1-GGUF 的 Q8_0 GGUF(由 Aleph Alpha 的 FP8 检查点转换而来;近乎无损)。我们没有从 156 GB 的 BF16 权重转换,因此本卡片上每个数据的参照都是该 Q8_0。重要性矩阵由我们自己(llama-imatrix)在德语和英语文本、代码以及模型自己带思考的聊天回答上计算。
2. 每个大型权重矩阵针对每种候选类型(Q2_K、IQ2_XXS、IQ2_XS、IQ2_S、IQ3_XXS、IQ3_S、Q3_K、IQ4_XS、Q4_K、Q8_0)使用 llama-quantize 和重要性矩阵各量化一次;每种选择的误差按矩阵估计(按重要性加权),并乘以一个 用真实 KL 散度测得的敏感度系数:每组张量(例如 FFN down 投影、注意力 value 投影、首尾几层)被单独移到较低比特的类型,并将它造成的 KL 散度与其量化误差进行比较。
3. 精确的预算分配(以字节为单位的多选背包问题)为每个目标大小为每个矩阵挑选一种类型;最终模型由已存储的张量组装而成,无需重新量化。 我们不公布逐张量的选择。归一化层和小张量保持高精度。

局限

• 仅用短的留出文本上的 KL 散度和困惑度测量,没有使用下游基准测试;请勿将其理解为任务质量的声明。
• 量化总会损失一些质量;最小的文件损失最大。
• 在上游支持落地之前,需要打过补丁的 llama.cpp(见顶部横幅)。本卡片上的所有数据都是用该构建(Vulkan 后端)测得的。
• 移除了什么: 在每一层中,移除被我们的校准文本(德语和英语文本、代码,以及模型自己带思考的聊天回答)选中次数最少的 19 个路由专家。在这些文本上,只有约 0.07 % 的专家选择落在被移除的专家上(任何一层最多 0.23 %);被移除的 950 个专家中有 375 个从未被选中。裁剪是对已量化的专家张量、路由器权重和路由器偏置进行切片,没有重新量化,也没有再训练;top-6 路由不变。这些文件是 Sakura-MicroQuality-Kolibri-1-GGUF 中对应文件的逐字节切片。我们的文本未覆盖的内容可能依赖被移除的专家,请检查你自己的使用场景。
• 与从 BF16 转换相比,从 Q8_0 源文件重新量化会增加极小的额外误差;质量列将每个文件与完整模型的同一个 Q8_0 源文件进行比较。
• 模型卡片推荐的设置:temperature 1.0、top-p 0.97、top-k 128;推理强度按请求设置(none、low、medium、high)。

致谢

• 原始模型:Kolibri-1 (Aleph Alpha, Apache-2.0),Apache-2.0 许可证(以 LICENSE 随附)。
• Q8_0 源文件和 llama.cpp 补丁:Hob-forge/Kolibri-1-GGUF(Hob-forge / Seraphiel102)。参照量化:完整模型的 llama-quantize 默认配方,配以我们的重要性矩阵。
• GGUF 格式和工具:ggml-org/llama.cpp (MIT)。