Ling-3.1-flash-1019 — EXL3 1.5/1.0 bpw (mixed)

EXL3 quant of inclusionAI/Ling-3.1-flash-1019 (560B total / 25B active MoE, 1M-token context), produced locally with exllamav3 1.6.6 on 5× RTX 3090.

This variant exists to do the one thing the 2.0 bpw build cannot: run without any CPU-resident experts, so decode stops paying the host round-trip. Routed experts are split per layer — half the MoE layers at 1.5 bpw, half at 1.0 — for an average of 1.25 bpw on the 97% of the weights that are experts.
Quant method EXL3 (exl3, trellis, mul1 codebook, out_scales auto)
Average bitrate ~1.29 bpw over the whole model (1.25 on routed experts)
Size on disk ~99.3 GB / 92.4 GiB tensors
Calibration 250 rows × 2048 cols, exllamav3 standard cal data
MTP 1 nextn layer, 4 bpw (draft_mode: mtp works)
Converted 2026-11-01, exllamav3 1.6.6
Base model licence MIT

Why per-layer and not per-expert

BlockSparseMLP packs all 512 experts of a layer into one MultiLinear and exllamav3 asserts a single bitrate per layer (multilinear.py:26), so a half-and-half split inside a layer will not load. The 50:50 split is therefore across layers, which keeps the average bitrate identical.

Which 34 layers get the extra half bit comes from the per-tensor proxy_err recorded during the 2.0 bpw conversion — the layers whose experts quantize worst (~1.53× spread between layers) are the ones promoted to 1.5:

6–20, 43–45, 48–53, 55–64

Within-layer expert error spread is only ~1.17×, so layer granularity gives up little compared with a hypothetical per-expert mix.

Per-tensor bitrate

Component bpw
Routed experts — 34 MoE layers 1.5
Routed experts — 34 MoE layers 1.0
Shared experts 5.0
KDA / linear-attention projections 4.0
Dense MLP (layers 0–3) 4.0
Gated MLA (9 layers) 6.0
lm_head 6.0
MTP layer (72) + its experts 4.0
Embeddings, norms, router, KDA convs / b_proj unquantized

Recipe included as recipe/ling-1.5-1.0bpw.yaml (105k entries). Everything outside the routed experts is byte-identical in bitrate to the 2.0 bpw build.

Context budget

The hybrid architecture makes context nearly free: only 9 of 72 layers keep a per-token KV, and MLA stores a 512-value latent + 64-value RoPE key, which is 416 B/token/layer at Q4 (1152 B fp16). The other 63 layers are KDA linear attention with a fixed-size recurrent state.
64K ctx 0.24 GiB
128K ctx 0.47 GiB
256K ctx 0.91 GiB
Q8 KV at 256K 1.47 GiB

The recurrent state pool is ~1.2 GiB regardless of context length; recurrent checkpoints live in system RAM (sysmem_recurrent_cache, default 4 GB). So the full 262,144-token pool fits comfortably — the spare VRAM goes to concurrency and load-time margin, not context.

Requirements

EXL3: exllamav3 ≥ 1.6.6 (BailingMoeV3ForCausalLM / bailing_hybrid). Works standalone or through TabbyAPI. Not usable with transformers / vLLM / llama.cpp.

Weight budget on 5× 24 GB (≈117 GB usable, minus ~3 GB for a resident STT service): ~92.4 GiB weights + ~5 GiB workspace/dequant transient + ~1–4 GiB cache → no CPU expert offload needed.

Usage (TabbyAPI)

tabby_config.yml in this repo is the intended configuration:

model:
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: "8,8"
  max_batch_size: 8
  tensor_parallel: false
  gpu_split_auto: false
  gpu_split: [22.5, 23.0, 23.0, 20.0, 23.0]   # GPU3 also hosts parakeet (~2.8 GB)
  chunk_size: 2048
  warmup: true
  template_vars_default:
    thinking_option: "off"
  sampling:
    temperature: {override: 0.7, force: false}
    top_p: {override: 0.9, force: false}
    top_k: {override: 40, force: false}
    min_p: {override: 0.05, force: false}
draft_model:
  draft_mode: mtp
  draft_num_tokens: 2
  hybrid_draft_tokens: 64
  dynamic_draft: true

Q8 KV was 25–70% faster than Q4 in the sweep because it lifts MTP acceptance from ~42% to 63–95%, and it costs only ~0.8 GiB at 256K.

Performance

Measured on 5× RTX 3090, exllamav3 1.6.6, with a 2.8 GB STT service resident on GPU 3. 256 output tokens, temperature 0; “cold” = uncached prompt, “warm” = same prefix again.

probe cold TTFT warm TTFT decode T/s
12K prompt, 1 stream 9.6 s 0.5 s 57.5
12K prompt, 4 streams 53–57 s 1.4 s 24–31 each
12K prompt, 8 streams 111 s 3.1 s 16–22 each
48K prompt, 1 stream 59.8 s — 50.0
128K prompt, 1 stream 134 s — 38.2
256K prompt, 1 stream 494 s — 27.7

Realistic prose at ~25K context: 39 T/s, MTP acceptance 63–65%. Repetitive filler reaches 54 T/s (95% acceptance) — an upper bound, not a promise. No CPU expert offload at any point. Full sweep: SWEEP.md in this repo.

Cold concurrency is prefill-bound: N simultaneous 12K prompts cost roughly N × 10–14 s to first token because the engine serializes prefills. Warm, append-only conversations (the agent loop) do not pay this.

Files

config.json  tokenizer*.json  chat_template.jinja
configuration_bailing_moe_v3.py  modeling_bailing_moe_v3.py   (unused by exllamav3, kept for reference)
model-00001-of-000NN.safetensors ... model.safetensors.index.json
quantization_config.json
tabby_config.yml                 # intended 5×3090 config
recipe/ling-1.5-1.0bpw.yaml      # per-tensor bitrate recipe used

Caveats

  • Converted from the FP8 release with exllamav3's standard calibration data. Quality was spot-checked for coherence (prose, JSON, instruction following); it has not been run against a benchmark suite or formally compared with the 2.0 bpw build.
  • 1.0 bpw experts are aggressive. Expect the degradation to show up in long multi-step reasoning and unusual languages first.
  • cache_size is 262,144 tokens, not 1M: at 1.33 bpw the remainder goes to KV/workspace and the box also runs an STT service. That is ~5 concurrent 48K sessions, or 8×24K.
  • The upstream model card's SGLang/vLLM instructions do not apply to this repo.
  • 2.0 bpw version (better quality, ~52 GB of experts on the CPU): auryn-macmillan/Ling-3.1-flash-1019-EXL3-2.0bpw

Credits

Base model: inclusionAI/Ling-3.1-flash-1019 (MIT). Quantization: exllamav3 by turboderp (MIT). Recipe derived from the ling-tp2 recipe published with the Ling-3.1 EXL3 release. Upstream README kept as UPSTREAM_README.md.