Ling-3.0-tiny — ASHQ1 Quantization (Top-Down)
ASHQ1 quantization of Ling-3.0-tiny (7.9B total / 1.3B active, 128 routed + 1 shared experts, KDA+MLA hybrid attention) with the top-down allocator: every tensor starts at F16 and is greedily downgraded cheapest-loss-first until the budget fits.
MoE-specific handling: routers (ffn_gate_inp, exp_probs_b) pinned at F16 outside the budget, shared experts mapped to high tiers (always active), routed experts graded by depth (early layers Q4_K → late layers Q6_K/Q8_0).
Note: File names contain "BF16" for HuggingFace parser compatibility — these are ASHQ1 quants, not BF16.
Quants
| File | Size | PPL (wiki.test.raw, ctx 1024) | Description |
|---|---|---|---|
Ling-3.0-tiny-BF16-ASHQ1-5000.gguf |
4861 MiB | 13.0681 ± 0.1051 | Bottom-up, best of the duel — smaller and better |
Ling-3.0-tiny-BF16-TOPDOWN-5000.gguf |
4999 MiB | 13.1094 ± 0.1054 | Top-down challenger (local only) |
Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf |
6013 MiB | 13.0001 ± 0.1047 | Top-down, first MoE quant |
Ling-3.0-tiny-BF16-TOPDOWN-6000-v7.1.gguf |
6016 MiB | 12.9802 ± 0.1045 | Top-down + phase-2 slack filling (local only) |
Ling-3.0-tiny-i1-Q6_K.gguf |
6198 MiB | 13.0123 ± 0.1049 | Uniform Q6_K with imatrix (reference, local only) |
Bottom-up vs top-down (@5000, same budget)
Two opposite allocations, same PPL within noise (Δ=0.04, σ=0.10):
| Tier | Bottom-up (4861 MiB) | Top-down (4999 MiB) |
|---|---|---|
| F16 | 215 / 9.7 MiB | 126 / 222.8 MiB |
| Q4_K | 43 / 2271 MiB | 210 / 2059 MiB |
| Q5_K | 30 / 2036 MiB | 36 / 2304 MiB |
| Q6_K | 8 / 94 MiB | 14 / 109 MiB |
| Q8_0 | 230 / 432 MiB | 140 / 285 MiB |
Bottom-up spreads upgrades wide (241 small tensors at Q8_0); top-down keeps the most precious tensors at F16 (incl. top-importance shared experts) and crushes the rest to the floor. Lesson: PPL can't separate the two paths — the bit distribution differs radically, the quality doesn't. Finer-grained tasks (HumanEval, MMLU) may.
Speed
MoE sparsity pays off: only 1.3B of 7.9B params active per token. On a GTX 1070 (8 GB) the quant runs at ~76 t/s mixed load and fits the full 131072-token context without problems.
Distribution (@6000, top-down)
| Tier | Tensors | Notes |
|---|---|---|
| F16 | 248 | pinned norms/routers + untouched high-importance |
| Q4_K | 183 | early routed experts, floors |
| Q5_K | 23 | incl. output/token_embd pins |
| Q6_K | 41 | late routed experts |
| Q8_0 | 31 | shared experts, MLA factors |
Usage
llama.cpp
llama-cli \
-m Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf \
--jinja \
-ngl 99 \
-c 8192
Recommended sampling: temperature 1.0, top_p 0.95, top_k 20.
Ollama
FROM ./Ling-3.0-tiny-BF16-TOPDOWN-6000.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 1.0
PARAMETER top_k 20
PARAMETER top_p 0.95
ollama create ling-tiny-topdown-6000 -f Modelfile
ollama run ling-tiny-topdown-6000
Quantization Config (@6000, top-down)
Generated by ASHQ1 top-down. Reproduce with:
python main.py --model Ling-3.0-tiny-bf16.gguf --imatrix Ling-3.0-tiny-imatrix.gguf --size 6000 --top-down --run
--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(23)\.ffn_down_exps=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_gate=Q8_0"
--tensor-type "(blk|BLK)\.(0)\.ffn_up=Q8_0"
--tensor-type "(blk|BLK)\.((?:2|3|4|5|6|7|8|9|10|11|12|13))\.ffn_down_shexp=Q8_0"
--tensor-type "(blk|BLK)\.(1)\.ffn_down_shexp=Q6_K"
--tensor-type "(blk|BLK)\.((?:6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21))\.attn_output=Q8_0"
--tensor-type "(blk|BLK)\.((?:17|18|19|20|21|22))\.ffn_down_exps=Q6_K"
--tensor-type "(blk|BLK)\.(0)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_exps=Q6_K"
--tensor-type "(blk|BLK)\.([2-5])\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.((?:9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_exps=Q6_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_up_exps=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ffn_gate_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23))\.ffn_down_shexp=F16"
--tensor-type "(blk|BLK)\.((?:12|13|14|15|16))\.ffn_down_exps=Q5_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11))\.ffn_down_exps=Q4_K"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.exp_probs_b=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_inp=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_beta=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_k=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_mqa=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_gate_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_b=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_q=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_f_a=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.attn_v=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_g_a=F16"
--tensor-type "(blk|BLK)\.((?:1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_up_shexp=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_gate=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_k_b=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_v_b=F16"
--tensor-type "(blk|BLK)\.((?:22|23))\.attn_output=F16"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_up_exps=Q5_K"
--tensor-type "(blk|BLK)\.([2-8])\.ffn_gate_exps=Q5_K"
--tensor-type "(blk|BLK)\.([0-1])\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_k=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.ffn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_a=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_dt=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23))\.attn_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_v=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_norm=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22))\.ssm_conv1d_q=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_q_a_norm=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23)\.attn_kv_a_norm=Q4_K"
--tensor-type ".*output_norm.*=Q4_K"
Notes
- Source BF16:
inclusionAI/Ling-3.0-tiny(24 layers,bailingmoe3arch, no MTP) - Imatrix: custom Ling-3.0-tiny imatrix
- MoE: 128 routed experts (8 active) + 1 shared expert per token, expert FF 512 (no K-quant padding needed)
- Quantization and tuning by wepiqx
- Built with llama.cpp
- License: Apache-2.0 (upstream)