Ling-3.1-flash — Wrench EXL3 2.10 bpw
A 2.10 bpw EXL3 quant of inclusionAI/Ling-3.1-flash, 153.6 GB. Runs with GPU weights and CPU experts in system RAM. Weighted toward agentic coding and tool use, with a general-text floor.
| Base model | Ling-3.1-flash (MIT); architecture |
| Runs with | exllamav3 0f8cf2c3f052a2503e98c1e77d40b2f055047bcc plus the published ROCm patch |
| Tested fit | R9700 32 GB, 192 GB host RAM (peak about 135 GiB system RAM and 31 GiB VRAM); 153.6 GB weights plus dependencies/caches; 128 GB RAM does not fit |
| Results, method, reproduce | Ling-3.1 page |
Results
| Context | Domain | Prefill tok/s | Decode tok/s | TTFT (s) |
|---|---|---|---|---|
| 8K | prose | 486.6 | 12.02 | 16.84 |
| 8K | code | 498.7 | 12.07 | 16.43 |
| 64K | prose | 476.5 | 7.86 | 137.53 |
| 64K | code | 476.4 | 7.64 | 137.58 |
Measured 2026-10-18: R9700, 9800X3D, 192 GB DDR5; ROCm 7.2.1, exllamav3 pin above plus patch; benchmark runner with
explicit device load (--device cuda:0), EXL3_MOE_STREAM_T=6, chunk 8192, 62 CPU-MoE layers (+38-43% prefill over the
default autosplit load; decode unchanged; no measurable quality change against FP8).
Greedy, 256 actual output tokens, no MTP/reuse; two runs after a warm-up.
Spreads: reproduction guide.
| Candidate | Size | KL top-64 | Top-1 |
|---|---|---|---|
| Wrench EXL3 2.10 bpw | 153.6 GB | 0.922 | 77.99 % |
| Best other public quant at this size (as of 2026-10-18) | 153.6 GB | 0.962 | 77.66 % |
| bartowski IQ2_XXS (GGUF) | 170.8 GB | 1.085 | 75.21 % |
Verdict. Against bartowski's IQ2_XXS GGUF (170.8 GB, 11% larger), Wrench is closer to the FP8 model on all 92 windows (KL 0.922 vs 1.085; 95% CI of the difference +0.138 to +0.189) with 2.8 points higher top-1; bartowski's file gives the actual next token higher probability on the three agentic-coding windows of six full-vocabulary windows. Against the best other public quant at this size, the result is mixed: Wrench is 4.1% closer by KL (95% CI -0.052 to -0.027), all on agentic-coding windows; on general text the other file is slightly closer, and it gives the actual next token higher probability (NLL 0.085 lower, 6/6). Repeated measurements were identical. KL is not task accuracy. Same-config repeat floor: 0.0000.
FP8 agreement on 92 × 4096-token windows; not task accuracy. Scored on the R9700 with exllamav3 in its deterministic settings. Method limits.
Run it
Known ROCm issue: first decode fails on shared_memory_per_block without the published patch.
Apply it using the install guide below.
Prepare the pinned venv, both patches and the environment in INSTALL
(the explicit-device patch
adds -dev to examples/chat.py). Set MODEL and EXL3 as there. The download and the interactive console loop are
untested; the same load measured 502.4 tok/s prefill / 12.24 decode on the public 8K code input.
hf download boxwrench/Ling-3.1-flash-exl3-wrench-2.10bpw --local-dir "$MODEL"
EXL3_MOE_STREAM_T=6 python "$EXL3/examples/chat.py" -m "$MODEL" -dev cuda:0 -nw -mode ling -no_think -temp 0 \
-mcl 62 -mct 8 -gcs 8192 -cs 9216
-dev skips exllamav3's VRAM budgeting, so run it on an idle card. After starting, send one short message before the first long prompt: with -nw, a cold first 8K request returned non-finite logits at step 0 (a known issue; a short request first avoids it). For an OpenAI-compatible local endpoint use
bench/serve_fast.py.
Other GPUs/CUDA are untested; the ROCm fallback is verified only on gfx1201.
For agents: AI_SETUP.
Showcases
One-shot three.js demos written by this quant on one R9700, thinking on, unedited: lighthouse islet at dusk and pixel pagoda (prompts, reasoning and run records).
Your numbers
Credits
inclusionAI for the base model (MIT); exllamav3 for conversion/runtime (MIT). Licence notices.