How to Run Ling 3.1 flash 1019 Locally

GGUF builds of inclusionAI's Ling 3.1 flash 1019, with the multi-token prediction head inside. The calibration corpora behind our builds are public.

• Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
• Runs in llama.cpp build b11571 or newer.
• The multi-token prediction head is inside the file: add --spec-type draft-mtp to use it. In our check 57 to 69% of its drafted tokens were accepted.

---

Pick a file

AD- marks an Atomic Dynamic layout: the routed experts (97% of the weights) take the low-bit type, while attention, the shared expert, the router, the dense layers and the multi-token prediction block stay at q8_0 or higher. The file name is the type of the experts' gate and up projections; their down projections, and all experts of the last five blocks, get a step more.

This file has not been measured against the BF16 model. The same layout on the previous checkpoint, with its KL divergence and top-1 agreement, and larger files are in AtomicChat/Ling-3.1-flash-GGUF.

How much memory it needs

The file has to fit, plus a little for the context. Context is cheap on this model: 63 of its 72 layers use linear attention with a fixed-size state, and only 9 keep a compressed attention cache. That is about 0.5 GB per conversation plus about 10 KB per token.

On a Mac, raise the GPU memory limit first, for example sudo sysctl iogpu.wired_limit_mb=240000 on a 256 GB machine.

Running it

llama-server -hf AtomicChat/Ling-3.1-flash-1019-GGUF:AD-IQ2_M -ngl 99 -c 32768 --jinja

The file is split into six shards of up to 45 GB. To download them yourself, take all six and point llama.cpp at the first:

hf download AtomicChat/Ling-3.1-flash-1019-GGUF --include "*AD-IQ2_M*" --local-dir .
llama-server -m Ling-3.1-flash-1019-AD-IQ2_M-00001-of-00006.gguf -ngl 99 -c 32768 --jinja

With the multi-token prediction head:

llama-server -hf AtomicChat/Ling-3.1-flash-1019-GGUF:AD-IQ2_M -ngl 99 -c 32768 --jinja --spec-type draft-mtp

The chat template is inside the file. Thinking is on by default. The template switches it with enable_thinking, so a system prompt alone does not turn it off; start the server with

llama-server -hf AtomicChat/Ling-3.1-flash-1019-GGUF:AD-IQ2_M -ngl 99 -c 32768 --jinja --chat-template-kwargs '{"enable_thinking": false}'

or send "chat_template_kwargs": {"enable_thinking": false} with a request.

How we built it

From inclusionAI's FP8 release (revision dddfee4b), converted to BF16 and quantized with llama.cpp b11571 (097cf33d9). Converting FP8 to BF16 rounds each weight once, far below what Q8_0 adds. The file is 3.17 bits per weight on average; we checked that it loads in llama-server and completes text and code.

The importance matrix is the one we computed for Ling 3.1 flash on 5.5M tokens of the ling3-moe corpus (thinking on and off, tool calls, 30 languages, code, long documents up to 32k tokens), rendered through the model's chat template. Ling 3.1 flash 1019 has the same architecture, configuration and chat template; we did not recompute the matrix for this checkpoint. The matrix and the layouts are in AtomicChat/Ling-3.1-flash-1019-GGUF-metrics.

Model details

Mixture of experts: 512 routed experts in each of 68 layers (the first 4 layers are dense), 8 routed plus 1 shared expert per token, about 24B parameters active of 564B, plus an 8B multi-token prediction block. Hybrid attention: 63 Kimi Delta Attention (linear) layers and 9 multi-head latent attention layers. Hidden size 4096, vocabulary 157,184, context 262,144. Needs llama.cpp with bailingmoe3 support.

License and credits

MIT, as the original. Base model Ling 3.1 flash 1019 by inclusionAI. Quantization by AtomicChat.