Sakura Qwen3.8-27B Humanlike Chat 2.0 — Uncensored AutoRound Q3_K_M GGUF

This repository provides quantized GGUF releases representing the synergy of three key components: Uncensored + Humanlike 2.0 + AutoRound Q3_K_M.

• Recommended / Winner: Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf (AutoRound GPU Q3_K_M)
• Comparison Baseline: Humanlike-Chat-2.0-Uncensored-imatrix-Q3_K_M.gguf (Standard llama.cpp Imatrix Q3_K_M)

Key Result: AutoRound Q3_K_M won the predefined Q3 fidelity gate against the same-size Imatrix Q3_K_M baseline.

---

Model Provenance

Qwen/Qwen3.8-27B
  ↓
huihui-ai/Huihui-Qwen3.8-27B-abliterated
  ↓
LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0
  ↓
Sakura AutoRound Q3_K_M

---

The Trio: Uncensored • Humanlike 2.0 • AutoRound

1. Uncensored (Abliterated Base): The foundation originates from huihui-ai/Huihui-Qwen3.8-27B-abliterated. Refusal vectors were removed through targeted layer-wise abliteration specifically across layers 18 to 51 (leaving initial representation layers 0 to 17 and vision/MTP untouched), effectively stripping refusal mechanisms while preserving core model reasoning and capabilities.
2. Humanlike 2.0 (Conversational Fine-Tune): Fine-tuned by LessThanThreeAI in LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 using on-policy distillation. The model is specifically tuned to text and converse naturally like a human, with nuance, personality, and roleplay capability—without requiring elaborate system prompt instructions.
3. AutoRound (Advanced 3-Bit Quantization): Quantized using Intel's AutoRound optimization across all 64 layers against BF16 reference logits. AutoRound significantly reduces layer-wise rounding error in low-bit representation, beating the same-size importance-matrix quant across held-out domains.

---

Model Details

• Direct Upstream Base: LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0
• Abliterated Ancestor: huihui-ai/Huihui-Qwen3.8-27B-abliterated
• Base Architecture: Qwen3.8-27B hybrid architecture with linear attention / SSM and full attention (64 transformer layers + 1 MTP layer)
• Context Length: 262,144 tokens (native context support in llama.cpp)
• License: Apache-2.0
• Intended Inference: Local CPU / GPU / Vulkan execution with llama.cpp

---

Why this Quant?

Quantizing large 27B hybrid models to 3-bit typically poses severe challenges to output fidelity. In this project, AutoRound optimization was run across the model's 64 layers against full-precision (BF16) reference logits to minimize layer-wise rounding error.

To ensure rigorous validation before publishing, AutoRound Q3_K_M was benchmarked head-to-head against a same-size standard importance matrix quant (Q3_K_M) using an identical BF16 baseline across three held-out test domains (English conversational text, developer code, and Wikipedia text).

The predefined release criterion required:

1. File size not larger than the baseline (within 1%)
2. Mean KL divergence improved by at least 10%
3. Better KL divergence on at least 2 of 3 evaluation texts

AutoRound GPU Q3 satisfied all three criteria and won the release gate.

---

Quantization Comparison

Evaluation protocol: llama-perplexity, context length 512, 12 chunks per text, batch size 512. Base logits evaluated in BF16 precision; quantized candidates evaluated with identical settings.

---

Important Benchmark Notes

For complete transparency:

• Mean KL divergence improved by approximately 15.8% (relative drop from 0.0463 to 0.0390).
• English conversational text improved (KL 0.0328 vs 0.0421; Top-1 96.4% vs 96.0%).
• Developer / code text improved significantly (KL 0.0332 vs 0.0470; Top-1 95.6% vs 94.5%).
• Wikipedia text regressed slightly (KL 0.0510 vs 0.0499; Top-1 90.0% vs 91.1%).
• Perplexity (PPL) was slightly worse for AutoRound on all three evaluation sets (EN: 1.734 vs 1.728; dev: 1.841 vs 1.840; Wiki: 7.680 vs 7.615). PPL was not the predefined gate metric because KL divergence and top-1 agreement directly measure fidelity to original model output logits.
• Size difference: AutoRound is 320 bytes smaller than the Imatrix file (13,500,736,832 vs 13,500,737,152 bytes), which is essentially identical size; 320 bytes is not a meaningful size benefit.
• The results apply specifically to the tested evaluation domains.

---

Files in this Repository

---

Download

Recommended: AutoRound Q3_K_M (Winner)

Using the Hugging Face CLI:

hf download webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf --local-dir .

Or via curl:

curl -L -o Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf \
  https://huggingface.co/webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF/resolve/main/Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf

Baseline: Imatrix Q3_K_M (For Comparison)

hf download webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF Humanlike-Chat-2.0-Uncensored-imatrix-Q3_K_M.gguf --local-dir .

---

llama.cpp Example

Run with llama-cli or llama-server:

llama-cli \
  -m Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  -p "<|im_start|>user\nHello! How can you assist me today?<|im_end|>\n<|im_start|>assistant\n"

Note on Thinking: Like the upstream Humanlike-Chat 2.0 model, thinking tokens are enabled by default in the chat template.

---

Related Sakura models

• webmp3/Sakura-Humanlike-Chat-2.0-JEV-AutoRound-GGUF: JEV-enabled variant with calibrated System-One decision capabilities.

---

License & Attribution

• Upstream Model: LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 by LessThanThreeAI.
• Abliterated Ancestor: huihui-ai/Huihui-Qwen3.8-27B-abliterated by huihui-ai.
• Base Architecture: Qwen3.8 by Alibaba Cloud.
• License: Released under the Apache-2.0 License.
• Quantization: Sakura Quantization Pipeline with Intel AutoRound & llama.cpp.

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura Qwen3.8-27B Humanlike Chat 2.0 — Uncensored AutoRound Q3_K_M GGUF

本仓库提供量化的 GGUF 发布版本,体现了三个关键要素的协同: Uncensored + Humanlike 2.0 + AutoRound Q3_K_M。

• 推荐 / 胜出: Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf(AutoRound GPU Q3_K_M)
• 对照基线: Humanlike-Chat-2.0-Uncensored-imatrix-Q3_K_M.gguf(标准 llama.cpp Imatrix Q3_K_M)

关键结果: AutoRound Q3_K_M 在预先设定的 Q3 保真度关卡中胜过了相同大小的 Imatrix Q3_K_M 基线。

---

模型谱系

Qwen/Qwen3.8-27B
  ↓
huihui-ai/Huihui-Qwen3.8-27B-abliterated
  ↓
LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0
  ↓
Sakura AutoRound Q3_K_M

---

三者组合:Uncensored • Humanlike 2.0 • AutoRound

1. Uncensored(去除拒绝的基础模型): 基础来自 huihui-ai/Huihui-Qwen3.8-27B-abliterated。拒绝向量通过有针对性的逐层 abliteration 被移除,具体作用于第 18 至 51 层(保留最初的表示层 0 至 17 以及视觉/MTP 不变),在保留模型核心推理能力的同时有效去除了拒绝机制。
2. Humanlike 2.0(对话微调): 由 LessThanThreeAI 在 LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 中通过 on-policy 蒸馏微调。该模型专门调校为像人一样自然地发消息和交谈,具有细腻的语气、个性和角色扮演能力,而无需复杂的系统提示指令。
3. AutoRound(先进的 3 比特量化): 使用 Intel 的 AutoRound 优化,针对 BF16 参照 logits 对全部 64 层进行量化。AutoRound 显著降低了低比特表示中的逐层舍入误差,在留出的各个领域上胜过了相同大小的重要性矩阵量化。

---

模型详情

• 直接上游基础模型: LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0
• abliterated 的祖先模型: huihui-ai/Huihui-Qwen3.8-27B-abliterated
• 基础架构: Qwen3.8-27B 混合架构,包含线性注意力 / SSM 和全注意力(64 个 Transformer 层 + 1 个 MTP 层)
• 上下文长度: 262,144 个 token(llama.cpp 原生支持的上下文)
• 许可证: Apache-2.0
• 预期推理方式: 使用 llama.cpp 在本地 CPU / GPU / Vulkan 上运行

---

为什么选这个量化?

把 27B 混合模型量化到 3 比特,通常会对输出保真度造成严重挑战。在本项目中,AutoRound 优化针对全精度(BF16)参照 logits 在模型的 64 层上运行,以最小化逐层舍入误差。

为确保发布前的严格验证,我们把 AutoRound Q3_K_M 与相同大小的标准重要性矩阵量化(Q3_K_M)进行了正面对比,使用相同的 BF16 基线,在三个留出的测试领域(英文对话文本、开发者代码和维基百科文本)上测量。

预先设定的发布标准要求:

1. 文件大小不超过基线(在 1% 以内)
2. 平均 KL 散度至少改善 10%
3. 在 3 个评测文本中至少 2 个上取得更好的 KL 散度

AutoRound GPU Q3 满足全部三项标准,通过了发布关卡。

---

量化对比

评测协议:llama-perplexity,上下文长度 512,每个文本 12 个块,批大小 512。基础 logits 以 BF16 精度评估;量化候选以相同设置评估。

---

重要的基准测试说明

为保证完全透明:

• 平均 KL 散度改善约 15.8%(从 0.0463 降到 0.0390 的相对降幅)。
• 英文对话文本有所改善(KL 0.0328 对 0.0421;Top-1 96.4% 对 96.0%)。
• 开发者 / 代码文本明显改善(KL 0.0332 对 0.0470;Top-1 95.6% 对 94.5%)。
• 维基百科文本略有退步(KL 0.0510 对 0.0499;Top-1 90.0% 对 91.1%)。
• AutoRound 在全部三个评测集上的 困惑度(PPL)略差(EN: 1.734 对 1.728;dev: 1.841 对 1.840;Wiki: 7.680 对 7.615)。PPL 不是预先设定的关卡指标,因为 KL 散度和 top-1 一致性直接衡量对原始模型输出 logits 的保真度。
• 大小差异: AutoRound 比 Imatrix 文件小 320 字节(13,500,736,832 对 13,500,737,152 字节),大小基本相同;320 字节并不构成有意义的大小优势。
• 这些结果仅适用于所测试的评测领域。

---

本仓库中的文件

---

下载

推荐:AutoRound Q3_K_M(胜出)

使用 Hugging Face CLI:

hf download webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf --local-dir .

或通过 curl:

curl -L -o Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf \
  https://huggingface.co/webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF/resolve/main/Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf

基线:Imatrix Q3_K_M(用于对比)

hf download webmp3/Sakura-Humanlike-Chat-2.0-AutoRound-GGUF Humanlike-Chat-2.0-Uncensored-imatrix-Q3_K_M.gguf --local-dir .

---

llama.cpp 示例

使用 llama-cli 或 llama-server 运行:

llama-cli \
  -m Humanlike-Chat-2.0-Uncensored-AutoRound-Q3_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  -p "<|im_start|>user\nHello! How can you assist me today?<|im_end|>\n<|im_start|>assistant\n"

关于思考的说明: 与上游的 Humanlike-Chat 2.0 模型一样,聊天模板中默认启用思考 token。

---

相关的 Sakura 模型

• webmp3/Sakura-Humanlike-Chat-2.0-JEV-AutoRound-GGUF:带有 JEV 的变体,具备经过校准的 System-One 决策能力。

---

许可证与归属

• 上游模型: LessThanThreeAI 的 LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0。
• abliterated 的祖先模型: huihui-ai 的 huihui-ai/Huihui-Qwen3.8-27B-abliterated。
• 基础架构: 阿里云的 Qwen3.8。
• 许可证: 以 Apache-2.0 许可证 发布。
• 量化: 使用 Intel AutoRound 和 llama.cpp 的 Sakura 量化流水线。