GGUF conversions of inclusionAI/Ling-3.0-flash (124B total / 5.1B active, hybrid KDA + gated MLA, 512-expert MoE), converted directly from the released BF16 safetensors.
These are the reference conversions for the bailingmoe3 architecture, merged into llama.cpp in
PR #26608 (2026-08-17). Every file bundles the MTP (NextN)
block and Ling 3.0's trained per-layer SwiGLU clamp metadata, and no separate drafter file, nor fork
required.
Consistent agentic use (tool calling, reasoning split) currently requires two llama.cpp PRs:
Without both, tool calls inside an unclosed think block are dropped and some turns fail with a 500.
To run with llama-server:
llama-server -hf bloomer010/Ling-3.0-flash-GGUF:Q4_K_S
Generally...
Larger files = More precision.
Smaller files = More compression = More slop and misbehavin'.
Weights and context share your memory, so be sure leave headroom.
| your memory | file | size |
|---|---|---|
| 192 GB+ | UD-Q8_K_XL |
177 GB |
| 128 GB | Q8_0 |
136 GB |
| 96 GB | UD-Q6_K_XL |
116 GB |
| 80 GB (A100/H100) | Q5_K_M |
92 GB |
| 64 GB | Q4_K_M |
78 GB |
| 56 GB | Q4_K_S / MXFP4_MOE¹ |
74 / 70 GB |
| 48 GB | Q3_K_M |
63 GB |
| 32 GB | UD-Q2_K_XL / IQ2_M |
43 / 42 GB |
| 24 GB | IQ1_M (with expert offload, see below) |
30 GB |
¹ MXFP4_MOE runs its native path on MXFP4-capable GPUs (Blackwell RTX 50-series, GB10/DGX
Spark). Elsewhere it falls back to a slower dequant path — prefer Q4_K_S on older hardware.
With less VRAM than the file size, keep the experts on CPU and the rest on GPU, e.g.:
llama-server -hf bloomer010/Ling-3.0-flash-GGUF:IQ1_M \
-ngl 99 -ot "ffn_.*_exps\.weight=CPU" -c 32768
Recommended sampling from the source model card: temperature 0.6, top_p 0.95, top_k 20.
Thinking mode is on by default; disable per request with
"chat_template_kwargs": {"enable_thinking": false}.
./build/bin/llama-server \
-m Ling-3.0-flash-Q4_K_S.gguf \
-c 262144 \
-ngl auto \
--flash-attn auto \
--temp 0.6 --top-p 0.95 --top-k 20 \
--jinja
🚨 Note that dspark was measured to be faster in at least one instance, and is likely faster on most setups. See the next section below.
Every quant bundles the MTP/NextN block. Enable it with --spec-type draft-mtp:
./build/bin/llama-server \
-m Ling-3.0-flash-Q8_0.gguf \
-c 262144 \
-ngl auto \
--flash-attn auto \
--temp 0.6 --top-p 0.95 --top-k 20 \
--jinja \
--spec-type draft-mtp
During ordinary inference, llama.cpp skips the MTP tensors and may report them as unused. With
--spec-type draft-mtp, the same GGUF is opened as an MTP draft model and block 42 is loaded and
executed. No separate drafter file is required.
Ling-3.0-flash-dspark-BF16.gguf (1.9 GB) and Ling-3.0-flash-dspark-Q4_K_M.gguf (665 MB) are speculative decoding drafts converted from inclusionAI/Ling-3.0-flash-dspark. Tested with llama.cpp as:
llama-server -m Ling-3.0-flash-Q5_K_M.gguf -md Ling-3.0-flash-dspark-Q4_K_M.gguf --spec-type draft-dspark --spec-draft-n-max 8
MoE placement can be adjusted for available VRAM with -ncmoe N. Draft-model placement can be controlled separately with -ncmoed N and -ngld N.
Supports up to 256K context.
Taken directly from the released inclusionAI/Ling-3.0-flash BF16 safetensors.
Conversion-specific tensor transformations include:
A_log stored as exp(A_log)kv_b_proj split into separate K and V tensors, with the K tensor transposedg_proj tensors mapped separatelyNorms, routing tensors, expert routing bias, KDA state scalars, dt_bias, and convolution weights
remain F32.
Importance matrix generated from the Q8_0 model:
wiki.train.rawMXFP4_MOE:
Q8_0:
UD-Q2_K_XL:
IQ1_S:
The GGUF contains 43 blocks:
The first two target layers use dense FFNs. The remaining target layers use 512 routed experts with top-8 selection plus one shared expert. Routing uses sigmoid scoring, expert bias, eight expert groups, and four selected groups.
The KDA safe gate is implemented as:
lower_bound * sigmoid(exp(A_log) * (f_proj(x) + dt_bias))
The lower bound is -5.0. The GGUF stores the positive exp(A_log) value, while the sign is
supplied by the negative lower bound.
git clone https://github.com/ggml-org/llama.cpp.git # bailingmoe3 merged 2026-08-17
# pre-merge builds:
# git clone --branch bailingmoe3-support https://github.com/aetherbird/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-cli llama-server