Sakura — Nanbeige4.2-3B (GGUF, three sizes)

Nanbeige4.2-3B is a compact reasoning and agent language model from the Nanbeige LLM Lab (about 4.2B parameters, 22 layers run twice in a loop, English and Chinese). This repository holds three GGUF files of different size made from the BF16 weights with our measured mixed-codec method. Pick one on the Files page; the table below says what each file is. Independent community quantization, not an official Nanbeige LLM Lab release. Part of the Sakura Mini line.

Quality at a glance

Same method, same texts, same machine for every file; lower KL divergence means closer to the BF16 original:

• Sakura-Nanbeige4.2-3B-1.66GiB.gguf vs llama.cpp default IQ3_XXS (1.66 GiB): KL divergence to the BF16 original 17 % lower on average over all 4 texts (same size).
• Sakura-Nanbeige4.2-3B-2.20GiB.gguf vs llama.cpp default IQ4_XS (2.21 GiB): KL divergence to the BF16 original 2 % higher on average over all 4 texts (0.02 GiB smaller).
• Sakura-Nanbeige4.2-3B-2.38GiB.gguf vs llama.cpp default Q4_K_M (2.40 GiB): KL divergence to the BF16 original 5 % lower on average over all 4 texts (0.02 GiB smaller).

Why: the bit budget per tensor is allocated from measured sensitivity (see How it was made), not from fixed rules. Full table below.

Which file should I take?

The three files

"Tensor types by size" lists the share of the file's bytes per ggml type (all are standard llama.cpp types, so the files run in stock llama.cpp). Quality columns: mean KL divergence of the quantized model's next-token distribution against the BF16 source on held-out text (lower is better) and how often the most likely token is unchanged (higher is better).

Comparison at similar size

Same texts, same context, same chunks, measured by us with llama-perplexity (BF16 source = reference; PPL of the BF16 model: de: 3.938, en: 2.314, dev: 2.134, wiki: 23.541). The reference quants are: llama.cpp default recipe, same importance matrix (ours), measured the same way. Sorted by size.

Method: 12 chunks of 512 tokens per text; texts: German held-out text, English held-out text, developer-text held-out set, English encyclopedia held-out text (not used in any calibration). Single measurement on one machine; small KLD differences at similar size are not a quality ranking.

How it was made

1. The BF16 GGUF was converted by us from the original Safetensors with the current convert_hf_to_gguf.py of llama.cpp; the importance matrix was computed by us (llama-imatrix, 96 chunks of 512 tokens on a German/English text mix plus bartowski's public calibration text).
2. Every large weight matrix was quantized once per candidate type (Q2_K, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0) with llama-quantize and the importance matrix; the error of each choice was estimated per matrix (importance-weighted) and scaled by a sensitivity factor measured with real KL divergence: every group of tensors (for example the FFN down projections, the attention value projections, the first and last layers) was moved alone to a lower-bit type, and the KL divergence it caused was compared with its quantization error.
3. An exact budget allocation (multiple-choice knapsack over bytes) picked one type per matrix for each target size; the final model was assembled from the stored tensors without re-quantizing. We do not publish per-tensor choices. Norms, small tensors and the multi-token-prediction layers stay at high precision.

Limits

• Measured only with KL divergence and perplexity on short held-out texts and not with downstream benchmarks; do not read it as a task-quality claim.
• Quantization always costs some quality; the smallest file costs the most.
• Each of the three files was allocated with the better of two calibration runs (probe texts German + English, or code + English); for the middle file the developer-text column is therefore not held out. The German text (small and large file) and the encyclopedia text (all files) are held out.
• The model runs its 22 layers in two loops; the loop count is stored in the GGUF and needs a llama.cpp build with the nanbeige architecture (official b11259 and newer).

Credits

• Original model: Nanbeige4.2-3B (Nanbeige LLM Lab, Apache-2.0), Apache-2.0 license (included as LICENSE).
• BF16 conversion and importance matrix: own work (llama.cpp). Reference quants: llama-quantize default recipes (IQ3_XXS, IQ4_XS, Q4_K_M) with the same importance matrix.
• GGUF format and tools: ggml-org/llama.cpp (MIT).

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura — Nanbeige4.2-3B(GGUF,三种大小)

Nanbeige4.2-3B 是 Nanbeige LLM Lab 推出的一个紧凑的推理与智能体语言模型(约 4.2B 参数,22 层循环运行两次,支持英语和中文)。本仓库包含 三个不同大小的 GGUF 文件,由 BF16 权重通过我们实测得出的混合编码方法制作。请在 Files 页面选择其中一个;下表说明了每个文件的用途。 这是独立的社区量化版本,不是 Nanbeige LLM Lab 的官方发布。属于 Sakura Mini 系列。

质量概览

每个文件使用相同的方法、相同的文本、同一台机器;KL 散度越低,越接近 BF16 原始模型:

• Sakura-Nanbeige4.2-3B-1.66GiB.gguf 对比 llama.cpp 默认的 IQ3_XXS (1.66 GiB):相对于 BF16 原始模型,在全部 4 个文本上平均 KL 散度 低 17 %(大小相同)。
• Sakura-Nanbeige4.2-3B-2.20GiB.gguf 对比 llama.cpp 默认的 IQ4_XS (2.21 GiB):相对于 BF16 原始模型,在全部 4 个文本上平均 KL 散度 高 2 %(小 0.02 GiB)。
• Sakura-Nanbeige4.2-3B-2.38GiB.gguf 对比 llama.cpp 默认的 Q4_K_M (2.40 GiB):相对于 BF16 原始模型,在全部 4 个文本上平均 KL 散度 低 5 %(小 0.02 GiB)。

原因:每个张量的比特预算是根据实测的敏感度分配的(见 制作方式),而不是按固定规则。完整表格见下文。

我该选哪个文件?

这三个文件

“各张量类型(按大小)”列出每个 ggml 类型占文件字节数的比例(全部是标准的 llama.cpp 类型,因此这些文件可在原版 llama.cpp 中运行)。质量列:量化模型的下一个 token 分布相对于 BF16 源文件在留出文本上的平均 KL 散度(越低越好),以及最可能的 token 保持不变的频率(越高越好)。

相近大小的比较

相同的文本、相同的上下文、相同的分块,由我们使用 llama-perplexity 测量(BF16 源文件 = 参照;BF16 模型的 PPL:de: 3.938,en: 2.314,dev: 2.134,wiki: 23.541)。参照量化为:llama.cpp 默认配方、相同的重要性矩阵(我们的)、以相同方式测量。按大小排序。

方法:每个文本取 12 个块,每块 512 个 token;文本为:德文留出文本、英文留出文本、开发者文本留出集、英文百科留出文本(未用于任何校准)。这是在一台机器上的单次测量;相近大小下 KLD 的细小差异并不构成质量排名。

制作方式

1. BF16 GGUF 由我们用 llama.cpp 当前的 convert_hf_to_gguf.py 从原始 Safetensors 转换而来;重要性矩阵由我们自己(llama-imatrix,在德/英混合文本加 bartowski 的公开校准文本上共 96 个块,每块 512 个 token)计算。
2. 每个大型权重矩阵针对每种候选类型(Q2_K、IQ2_S、IQ3_XXS、IQ3_S、Q3_K、IQ4_XS、Q4_K、Q5_K、Q6_K、Q8_0)使用 llama-quantize 和重要性矩阵各量化一次;每种选择的误差按矩阵估计(按重要性加权),并乘以一个 用真实 KL 散度测得的敏感度系数:每组张量(例如 FFN down 投影、注意力 value 投影、首尾几层)被单独移到较低比特的类型,并将它造成的 KL 散度与其量化误差进行比较。
3. 精确的预算分配(以字节为单位的多选背包问题)为每个目标大小为每个矩阵挑选一种类型;最终模型由已存储的张量组装而成,无需重新量化。 我们不公布逐张量的选择。归一化层、小张量和多 token 预测层保持高精度。

局限

• 仅用短的留出文本上的 KL 散度和困惑度测量,没有使用下游基准测试;请勿将其理解为任务质量的声明。
• 量化总会损失一些质量;最小的文件损失最大。
• 三个文件各自使用两次校准运行中较好的一次进行分配(探测文本为德语 + 英语,或代码 + 英语);因此对于中间的文件,开发者文本一列并非留出数据。德文文本(小文件和大文件)以及百科文本(所有文件)是留出的。
• 该模型把它的 22 层循环运行两次;循环次数存储在 GGUF 中,需要带有 nanbeige 架构的 llama.cpp 构建(官方 b11259 及更新版本)。

致谢

• 原始模型:Nanbeige4.2-3B (Nanbeige LLM Lab, Apache-2.0),Apache-2.0 许可证(以 LICENSE 随附)。
• BF16 转换和重要性矩阵:自己的工作(llama.cpp)。参照量化:llama-quantize 默认配方(IQ3_XXS、IQ4_XS、Q4_K_M),使用相同的重要性矩阵。
• GGUF 格式和工具:ggml-org/llama.cpp (MIT)。