Ling-3.1-flash-1019 — EXL3 3.03 bpw (Bertholomus AI)

EXL3 quantization (3.03 bits per weight) of inclusionAI/Ling-3.1-flash-1019 at revision dddfee4b, made by Bertholomus AI. Model and weights by inclusionAI, under the MIT License. This repository holds quantized weights only; the original model card, benchmarks and prompt format are on inclusionAI's model page. The chat template, the tokenizer files and the two Python files are inclusionAI's, unchanged.

Made to serve on two NVIDIA DGX Spark (GB10) nodes, tensor-parallel 2, with TensorFold (by ashhart) and our bailing_v3 engine family on branch ling-3.1-tp2. Recipe, launch and measurements: bertholomus/ling-3.1-tensorfold-tp2-2xgb10 (v0.2.0).

Quantization


Source	`inclusionAI/Ling-3.1-flash-1019` FP8, revision `dddfee4b`, written to a BF16 copy first (each FP8 weight times its block scale in fp32, rounded once to bf16)
Converter	exllamav3 1.5.3 by turboderp plus our Ling-3.1 module: branch `ling-3.1` of bertholomus/exllamav3, commit `13463cc9`; one pass of 20 h 7 min on one GB10, a second GB10 quantizing part of each layer from layer 8 on (a remote worker that gives the same bytes as a local run)
Command	`convert.py -rcp ling-tp2.yaml -hb 6 -mb 4`, with the recipe file from the recipe repository's `conversion/`, the same as for Ling-3.1-flash
Routed experts (68 layers, 512 each)	3 bits
Shared experts	5 bits
KDA attention (63 layers)	4 bits
Dense MLP (layers 0–3)	4 bits
MLA attention (9 layers)	6 bits; `kv_b_proj` unquantized
Output head (`-hb`)	6 bits
MTP layer (`-mb`)	4 bits
Codebook and calibration	mul1; the converter's default calibration, 250 rows × 2,048 tokens
Unquantized tensors	the embedding, norms, router, `kv_b_proj`, the KDA short convolutions and the rest the converter skips, written back after the conversion with the BF16 copy's exact bytes and dtype
Whole model	3.03 bpw by the converter's count, 35 shards, 220.6 GB

Quality

Not yet measured against the original FP8 model. What we measured is the engine's arithmetic on these weights: at every position of 40 prompts (66,103 positions), the engine's next-token logits against a reference forward of the same weights in fp32 (our PyTorch implementation written from inclusionAI's modeling code). Top-1 agreement 0.9712 and mean KL 0.0132, about what the same reference gives with bf16 activations (0.9726 and 0.0136); against that bf16 reference, the engine's top-1 agreement is 0.9715 and its mean KL 0.0138. The recipe has the full table.

Serving (measured, 2× GB10, TP2)

Engine commit 72f16e48, the 262,144-token window of the model's config, MTP drafting on, one request at a time, greedy:

Decode: 35.1–39.3 tok/s drafted on prompts up to 6.6K tokens (22.5–24.1 serial), 28.8–29.1 at 255K.
Prompt: 568–696 tok/s from 2K to 36K tokens, 224 at 255K (19.0 minutes to the first token).
Drafting is exact: drafted and serial decoding give the same tokens (6 of 6 test prompts).
Memory: the weights take 103.4 GiB of each node's 123.79 GiB, and a node keeps ~2-4 GiB free while serving.

Use

Download: hf download bertholomus/Ling-3.1-flash-1019-EXL3-3.03bpw --local-dir <EXL3_DIR>
Split it into one folder per rank (~112 GB each) with the engine's tools, as in step 5 of the recipe's Make the weights.
Start both ranks as in the recipe's Run it.

These are the files our served rank folders were split from, with this card as README.md.

Before you run this

Tested only with our TensorFold engine branch, on two GB10 nodes joined by a direct RoCE link. Our exllamav3 module converts the model, but its tensor-parallel export and the MTP draft attach are not written (they raise), and we have not run generation with exllamav3.
Little memory to spare. A node keeps only ~2-4 GiB free while serving (lowest 1.94 GiB in our checks), so nothing else can use the GPU.
262,144 tokens is the window we run. It is the config's max_position_embeddings; inclusionAI's model card names a 1M-token window, which we have not run.

Credits

inclusionAI — Ling-3.1-flash-1019 and its weights (MIT). These weights are a quantized derivative of theirs and stay under the MIT License. modeling_bailing_moe_v3.py here is theirs, unchanged (Apache-2.0, Copyright 2025 Antgroup and The HuggingFace Inc. team).
turboderp — EXL3 and exllamav3 (MIT): the format and the converter.
ashhart — TensorFold (Apache-2.0): the serving engine.
flash-linear-attention (fla-org, MIT) — the KDA kernels of our reference forward.
Bertholomus AI (Albert Lee, bertholomus) — the conversion, the Ling module for exllamav3, the bailing_v3 engine family and the measurements.

Not affiliated with or endorsed by inclusionAI, NVIDIA, the TensorFold authors or the exllamav3 authors.

bertholomus/Ling-3.1-flash-1019-EXL3-3.03bpw