Ling-3.0-tiny
English | 📖 中文文档
Abliterated APEX GGUF quants of inclusionAI/Ling-3.0-tiny — 7.9B total / 1.3B active hybrid-reasoning MoE
🔓 AbliterationRefusal direction removed with abliterix using LoRA steering search on a bnb-4bit loaded model. Over 130 candidate steering configurations were explored; the Pareto-selected recipe was merged into the BF16 weights.
85% fewer refusals (15/100 vs 98/100 baseline) at 0.0677 KL divergence. Post-abliteration capability spot-checks (math, logic, coding, translation, knowledge) all pass with no degradation observed.
💡 What is APEX?These GGUF files are quantized using APEX, a novel MoE-aware mixed-precision quantization technique that outperforms standard quantization methods while being significantly smaller.
APEX beats Q8_0 perplexity at half the size — and even beats F16.
APEX classifies every tensor by its role — routed expert, shared expert, or attention — and applies a layer-wise precision gradient, giving the most sensitive edge layers higher precision and compressing the redundant middle layers more aggressively. Ling-3.0-tiny's 128 routed experts (only 8 active per token) make it an ideal candidate.
📦 APEX Quantization Tiers| File | Size | BPW | Best For |
|---|---|---|---|
*-APEX-I-Quality.gguf | 5.77 GB | 5.84 | Highest quality, best accuracy |
*-APEX-I-Balanced.gguf | 5.96 GB | 6.03 | Best all-rounder, recommended |
*-APEX-I-Compact.gguf | 3.99 GB | 4.10 | Best quality/size ratio, 8 GB GPUs |
*-APEX-Mini.gguf | 3.41 GB | 3.45 | Smallest viable, 6 GB GPUs, long context |
All tiers quantized from a single BF16 source (15.07 GB) with the same diverse imatrix. Expert tensor layout: edge layers (L0–4, L19–23) keep higher precision than middle layers (L10–13); shared experts stay at Q8_0; the router is never quantized.
📐 I-Variant: Diverse Imatrix CalibrationAll tiers use a diverse calibration dataset spanning chat, code, reasoning, and tool-calling — no Wikipedia (500 chunks). This produces higher accuracy on real-world benchmarks, lower KL divergence, and only a tiny perplexity increase on wikitext.
🧠 Model Details| Architecture | BailingMoeV3 — hybrid KDA/MLA linear-attention MoE |
| Parameters | 7.9B total, 1.3B active per token |
| Experts | 128 routed experts + 1 shared expert, 8 routed active per token |
| Layers | 24 layers, 3:1 KDA–MLA stacking |
| Context | 131,072 tokens native |
| Reasoning | Native hybrid reasoning (thinking mode on by default) |
| Abliteration | abliterix LoRA steering (85% fewer refusals, 0.0677 KL) |
llama.cpp
hf download SC117/Ling-3.0-tiny-abliterated-APEX-GGUF --include "*.gguf" --local-dir ./models ./llama-server -m ./models/Ling-3.0-tiny-abliterated-APEX-I-Balanced.gguf -ngl 99 -c 32768 --flash-attn on --jinja
⚠️ bailingmoe3 architecture support: BailingMoE3 (PR #26608) was merged into llama.cpp master on 2026-08-17 — the first release containing it is b10470. Use llama.cpp b10470 or newer. If you see unknown model architecture: 'bailingmoe3', your build is too old — update and it will load.
Ollama
echo 'FROM ./Ling-3.0-tiny-abliterated-APEX-I-Balanced.gguf' > Modelfile ollama create ling-tiny-abliterated -f Modelfile && ollama run ling-tiny-abliterated
🎛️ Recommended Settings| Mode | Parameters |
|---|---|
| Thinking (default) | temp=1.0, top_p=0.95, top_k=20 |
| Fast / instruct | temp=0.7, top_p=0.8, top_k=20 (disable thinking via chat template) |
Links
- Original Model: https://huggingface.co/inclusionAI/Ling-3.0-tiny
- APEX Quantization: https://github.com/mudler/apex-quant
- abliterix: https://github.com/wuwangzhang1216/abliterix
- BailingMoE3 GGUF support (llama.cpp): https://github.com/ggml-org/llama.cpp/pull/26608 (merged 2026-08-17, first release b10470)
Citation
@misc{ling3tiny,
title = {{Ling-3.0-tiny}: A Lightweight Hybrid Reasoning MoE Model},
url = {https://huggingface.co/inclusionAI/Ling-3.0-tiny},
author = {{inclusionAI}},
year = {2026}
}