Ling-3.1-flash — EXL3 3.03 bpw (Bertholomus AI)

EXL3 quantization (3.03 bits per weight) of inclusionAI/Ling-3.1-flash at revision 7b864daa, made by Bertholomus AI. Model and weights by inclusionAI, under the MIT License. This repository holds quantized weights only; the original model card, benchmarks and prompt format are on inclusionAI's model page. The chat template, the tokenizer files and the two Python files are inclusionAI's, unchanged.

Made to serve on two NVIDIA DGX Spark (GB10) nodes, tensor-parallel 2, with TensorFold (by ashhart) and our bailing_v3 engine family on branch ling-3.1-tp2. Recipe, launch and measurements: bertholomus/ling-3.1-tensorfold-tp2-2xgb10.

Quantization

Quality

Not yet measured against the original FP8 model. What we measured is the engine's arithmetic on these weights: at every position of 40 prompts (66,103 positions), the engine's next-token logits against a reference forward of the same weights in fp32 (our PyTorch implementation written from inclusionAI's modeling code). Top-1 agreement 0.9723 and mean KL 0.0140, about what the same reference gives with bf16 activations (0.9728 and 0.0123). The recipe has the full table.

Serving (measured, 2× GB10, TP2)

Engine commit ef914199, a 196,608-token window (192K), MTP drafting on, one request at a time, greedy:

• Decode: 39.8–47.9 tok/s drafted on prompts up to 8K tokens (23.6–24.1 serial), 40.3 at 31K, 32.2 at 122K and 29.8 at 191K.
• Prompt: 595–703 tok/s from 1K to 8K tokens, 587 at 31K, 359 at 122K and 279 at 191K (11.4 minutes to the first token).
• Drafting is exact: drafted and serial decoding give the same tokens (6 of 6 test prompts).
• Memory: the weights take 104 GiB of each node's 123.79 GiB, and a node keeps ~2 GiB free while serving.

Use

1. Download: hf download bertholomus/Ling-3.1-flash-EXL3-3.03bpw --local-dir <EXL3_DIR>
2. Split it into one folder per rank (~112 GB each) with the engine's tools, as in step 5 of the recipe's Make the weights.
3. Start both ranks as in the recipe's Run it.

These are the files our served rank folders were split from, with this card as README.md.

Before you run this

• Tested only with our TensorFold engine branch, on two GB10 nodes joined by a direct RoCE link. Our exllamav3 module converts the model, but its tensor-parallel export and the MTP draft attach are not written (they raise), and we have not run generation with exllamav3.
• Little memory to spare. A node keeps only ~2 GiB free while serving (lowest 1.84 GiB), so nothing else can use the GPU.
• 192K is the ceiling with drafting. The model's config allows 262,144 positions; 262,144 with MTP drafting does not fit on two GB10.

Credits

• inclusionAI — Ling-3.1-flash and its weights (MIT). These weights are a quantized derivative of theirs and stay under the MIT License. modeling_bailing_moe_v3.py here is theirs, unchanged (Apache-2.0, Copyright 2025 Antgroup and The HuggingFace Inc. team).
• turboderp — EXL3 and exllamav3 (MIT): the format and the converter.
• ashhart — TensorFold (Apache-2.0): the serving engine.
• flash-linear-attention (fla-org, MIT) — the KDA kernels of our reference forward.
• Bertholomus AI (Albert Lee, bertholomus) — the conversion, the Ling module for exllamav3, the bailing_v3 engine family and the measurements.

Not affiliated with or endorsed by inclusionAI, NVIDIA, the TensorFold authors or the exllamav3 authors.