antirez-recipe quants
MoE quants following the antirez recipe: routed experts quantized hard, everything else protected.
| Tensor class | Quant | Notes |
|---|---|---|
blk.*.ffn_gate_exps, blk.*.ffn_up_exps (ffn_gate_up_exps if fused) |
Q5_K (imatrix) | routed-expert gate/up |
blk.*.ffn_down_exps |
Q8_0 (imatrix) | routed-expert down (K-quant for quality) |
blk.*.ffn_{gate,up,down}_shexp, all attn_*, output.weight |
Q8_0 |
shared experts / attention / head |
token_embd.weight |
F16 |
embeddings |
router (ffn_gate_inp), norms, sinks, biases |
source precision (BF16/F32) |
never quantized by llama.cpp |
Base type Q8_0 + --tensor-type overrides, imatrix calibration
(see SPEC per-model imatrix links). Runs on upstream llama.cpp — no fork or
patch needed.
Validation per file: gguf-dump tensor audit (experts = GU/DOWN, protection intact),
llama-cli smoke load, llama-perplexity spot ppl + --kl-divergence vs BF16 logits.
Files
| File | Size | SHA-256 |
|---|---|---|
Ling-3.0-tiny-Q4K-MXFP4-Q8.gguf |
5091030464 | 54859c79a6700c944b686bade6ef3ec062f1ca06c31a32fdeac8d2b350c870b4 |
Ling-3.0-tiny-Q5_K_M.gguf |
6898091232 | 34bd66526660e6d9c02b0f592cd2c0a95fc17b24b9345900d4d95061a8f18d07 |
Imatrix: https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF/Ling-3.0-tiny-imatrix.gguf
Recipe
Antirez recipe on inclusionAI/Ling-3.0-tiny (BF16 source: inclusionAI/Ling-3.0-tiny-GGUF, Ling-3.0-tiny-bf16.gguf):
ffn_gate_exps/ffn_up_exps-> Q5_Kffn_down_exps-> Q8_0- attention + shared experts + output -> Q8_0
- token embeddings -> F16
- imatrix: bloomer010/Ling-3.0-tiny-GGUF (
Ling-3.0-tiny-imatrix.gguf, sha-verified identical locally)
15,065 MiB BF16 -> 6,572 MiB (6.98 BPW).
Tokenizer note: Ling-3.0-tiny ships its own chat template (tokenizer.ggml.pre = bailingmoe2) — it works as-is on upstream llama.cpp, no extra flags needed.
Validation (llama.cpp, Ling arch)
- 12-case hard-smoke eval (temp 0, ctx 15360, full-resident
-ngl 999, AMD Vulkan/UMA): 4/12 PASS, ~120-124 t/s. Fails 2826, 10360, 6034, 10567, 9487, 1606, 1716, cwe121 (long-search loops that never commit a final answer; generations cut mid-thought). Passes 6500, 1618 and both big-code cases. Evaluated both raw and with the model's own template: generations are identical either way (4/12 both) — the template contributes nothing on this model. - Exact output tokens (own tokenizer) over the 4 solved: 26,017 = 6,504 tok/solved.
- KL divergence vs the official inclusionAI/Ling-3.0-tiny-GGUF Q8_0 (ds4 short prompts, ctx 60, 2 chunks, CPU): per-chunk KL 0.00995 / 0.01375 (avg 0.0119), delta-p RMS 3.3-3.8%, same-token top-1 96.6 / 94.8%.
- Eval parity with the Q4K-MXFP4-Q8 row (4/12 @ 124 t/s): this Q5_K_M build is a strictly higher-fidelity artifact of the same model (KL 0.0119, recipe-best class) at the same smoke score.