How to Run Ling 3.0 flash Sante Locally

Built from inclusionAI's original weights with our own importance matrix. The calibration corpora behind our builds are public.

• You can now run Ling in Atomic Chat.
• See our quantization analysis below for measurements and instructions.

---

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

AD- marks an Atomic Dynamic layout. The name says what the two largest tensor groups got: AD-<ffn_down>-<ffn_up>, collapsed to one name when both match. Nothing is named after a type it does not contain.

Which one fits your card

The file has to fit, and so does the context. This model keeps 560 KB of attention cache per token, which is 4.4 GB at 8k context and 17.5 GB at 32k. Budget for both.

[!TIP] If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words.

Running it

llama-server -m Ling-3.0-flash-Sante-AD-IQ4_XS.gguf -ngl 99 -c 8192

The chat template is inside every file, llama.cpp applies it on its own.

The model ships a multi token prediction head. It is inside every file here and needs no extra download:

llama-cli -m Ling-3.0-flash-Sante-AD-IQ4_XS.gguf --spec-type draft-mtp -ngl 99 -c 8192

Speed

How these compare to other builds

What we found while building these

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe ling3-moe. It is 4,685,467 tokens across 2,726 documents:

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

[!IMPORTANT] If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 3.1334 plus or minus 0.01199.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 144 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 8x A100-SXM4-40GB.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, in one run:

llama-imatrix -m Ling-3.0-flash-Sante-BF16.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special

Then, for AD-IQ4_XS:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.42\.=q8_0' \
  --tensor-type '^blk\.(2|3|39|40|41)\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.(2|3|39|40|41)\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_down_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_(gate|up)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_\w+\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ssm_\w+\.weight$=q8_0' \
  --output-tensor-type \
  f16 \
  --token-embedding-type \
  q8_0 \
  Ling-3.0-flash-Sante-BF16.gguf Ling-3.0-flash-Sante-AD-IQ4_XS.gguf IQ4_XS

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

42 layers, hidden size 2560, feed-forward size 6144, vocabulary 157,184, context 262,144. One multi token prediction head.

Needs a recent llama.cpp with bailingmoe3 architecture support. Built and tested with llama.cpp b11571 (097cf33d9).