Ling-3.1-flash-1019 — EXL3 2.06 bpw

EXL3 quant of inclusionAI/Ling-3.1-flash-1019 (560B total / 25B active MoE, 1M-token context), produced locally with exllamav3 1.6.6 on 5× RTX 3090.

This is the "small enough to actually run at home" tier: ~142 GiB of weights, of which ~90 GB stays in VRAM on five 24 GB cards and the cold tail of the routed experts lives in pinned host memory.

Per-tensor bitrate

The exact per-tensor recipe is included as recipe/ling-2.0bpw.yaml (105k entries, wildcard + exact keys as accepted by convert.py -rcp). The routed experts are ~97% of the weights, so 2.0 bpw on them is what sets the size.

Requirements

EXL3, so you need exllamav3 ≥ 1.6.6 (BailingMoeV3ForCausalLM / bailing_hybrid, Ling 3.x support landed in c0a19b1). Works with exllamav3 directly or through TabbyAPI. Not usable with transformers / vLLM / llama.cpp.

Weight budget:

• 141.6 GiB of EXL3 weights
• ~5 GB of K/V page pool + workspace at 64K context, Q4 cache
• so on 5× 24 GB (≈117 GB usable) about 52 GB of routed experts must live on the CPU — this is the config below.

RAM: ~55–60 GB for the pinned CPU-expert arena (plus 60 GB if you keep the FP8 source around).

Tested configuration (5× RTX 3090, TabbyAPI)

tabby_config.yml in this repo is the file that was actually used:

model:
  max_seq_len: 65536
  cache_size: 65536
  cache_mode: Q4
  max_batch_size: 4
  gpu_split_auto: false
  gpu_split: [19.0, 21.0, 21.0, 17.5, 21.0]
  cpu_moe_split_experts: 208   # per MoE layer, hot/cold promotion into spare slots
  cpu_moe_threads: 32
  warmup: true
draft_model:
  draft_mode: mtp
  draft_num_tokens: 5
  hybrid_draft_tokens: 64

Load: 100 s including warmup (26.9 s), from NVMe. GPU3 also had ~3.1 GB resident for a STT service, hence the smaller 17.5 GB ceiling there.

Measured performance (5× 3090, 2× Xeon, 247 GB RAM)

Decode, 600 tokens, MTP drafting on:

Concurrency (200 tokens/request):

Prefill: 12.5k tokens in 16.2 s (781 t/s), 51.8k tokens in 70 s (743 t/s), both uncached. Long-context prompts are CPU/host-bound in this offloaded configuration.

Sanity: needle retrieval at 30k and 58k tokens passed; output is coherent prose/JSON with the sampler in the bundled config (temp 0.7, top_p 0.9, top_k 40, min_p 0.05).

Usage (exllamav3, Python)

from exllamav3 import Model, Cache, Tokenizer, Generator, Config
from exllamav3.generator.sampler import ComboSampler

config = Config.from_directory("Ling-3.1-flash-1019-EXL3-2.0bpw")
model = Model.from_config(config)
cache = Cache(config, max_num_tokens=65536, cache_mode="Q4")
# 5× 24 GB: keep the tail of the experts on the CPU
model.load(gpu_split=[19.0, 21.0, 21.0, 17.5, 21.0], cpu_moe_split_experts=208)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model, cache, tokenizer, max_batch_size=4)

TabbyAPI: drop the directory under models/, keep the bundled tabby_config.yml next to it, and start TabbyAPI. If your cards are emptier than mine you can raise gpu_split / lower cpu_moe_split_experts — every expert pulled back to VRAM is ~3.7 MiB.

Files

config.json  tokenizer*.json  chat_template.jinja
configuration_bailing_moe_v3.py  modeling_bailing_moe_v3.py   (unused by exllamav3, kept for reference)
model-00001-of-00018.safetensors ... model-00018-of-00018.safetensors
model.safetensors.index.json  quantization_config.json
tabby_config.yml                 # tested 5×3090 config
recipe/ling-2.0bpw.yaml          # per-tensor bitrate recipe used

Notes / caveats

• Converted from the FP8 release with exllamav3's own calibration data, 250×2048 rows. No KL/perplexity comparison against FP8 was run; quality was checked with needle tests and manual generations, not a benchmark suite.
• The upstream model card's SGLang/vLLM instructions do not apply to this repo.
• 1M-token context is a property of the base model; this machine runs 64K. The MLA cache is small (kv_lora 512 + 64 rope dims per token, ~416 B/token at Q4), so more context is mostly a VRAM question.
• Also published: the 1.5/1.0 mixed variant that fits fully in VRAM on 5×3090 — see auryn-macmillan/Ling-3.1-flash-1019-EXL3-1.5-1.0bpw.

Credits

Base model: inclusionAI/Ling-3.1-flash-1019 (MIT). Quantization: exllamav3 by turboderp (MIT). Recipe derived from the ling-tp2 recipe published with the Ling-3.1 EXL3 release. Upstream README kept as UPSTREAM_README.md.