Flint in EXL3, 6.0 bits per weight

These are EXL3 weights of Recursive's Flint at 6.0 bits per weight, 30.5 GB in all. We converted them from the BF16 checkpoint, RecursiveSI/flint at revision 2a247be0, with exllamav3 1.6.2 and the patch in convert/, which adds the Flint architecture. Every projection is at 6 bits: attention, the two dense layers, the shared experts and the routed experts. The output head is at 8 bits. The router, the norms and the attention sinks stay at 16 bits.

This is the quality size. The 4.0 pack decodes about 38 % faster and loses more against BF16.

We serve these weights with TandemLLM, our inference engine for one DGX Spark. Flint has 49 layers. Two are dense; the other 47 have 256 routed experts, of which 6 run a token, plus one shared expert. Attention has 32 query heads and 4 KV heads with a learned sink per head. 35 layers use a 513-token sliding window with rotary, and 14 full layers use no positional encoding. The vocabulary has 150,144 tokens and the context is 262,144 tokens. One decoded token reads 2.0 GB of these weights.

One thing loaders need to know

The routed experts' SwiGLU activations get large in the top layers: |silu(gate) x up| reached 260,000 at layer 48 on a 2,048-token text, past fp16's 65,504. The converter's first run lost 131 of 250 calibration rows to infinities there. The patch keeps those activations in range: it stores every routed expert's up projection divided by 128 and multiplies the routing weights by 128. config.json says so: quantization_config.expert_interm_div is 128. A loader that ignores it gets expert outputs 128 times too small.

Quality against BF16

We compared the pack with the BF16 checkpoint on held-out text the model cannot have seen: TandemLLM's own docs and Python, written after Flint's training cutoff, plus wikitext-2. KL is the mean KL divergence from BF16 a position, and top-1 is how often both pick the same next token.

These numbers look worse than a 6-bit pack usually does, and the reason is the model, not the pack. On those 8 rows, two BF16 forwards of the same weights already differ by KL 0.034 and agree on 92.9 % of top-1, and an fp32 forward against a BF16 one is further apart: KL 0.18 and 91.4 %. Flint's router picks 6 of 256 experts by sigmoid scores that sit close together. One BF16 rounding of the residual stream changes the expert choice for 2 % of tokens at layer 2 and for 46 % by layer 46. The median KL stays near 0.01; a few positions carry the mean. The model_diff runs are in quality/.

Speed on a DGX Spark

One DGX Spark (GB10, 128 GB), nothing else running. TandemLLM 0.14.0. One request at a time, 512 greedy tokens, thinking off, a fresh server for each row. The game file row asks for a whole HTML game and runs 2,048 tokens. Edit and copy are asks whose answer repeats much of the prompt. Each cell is one run, in tok/s.

Speed is greedy with thinking off. A fresh TandemLLM install thinks at effort medium; docs/setup.md says how to change it and which effort each published number used. In every drafted run the text was the same as plain greedy, token for token. Flint has no draft layer, so TandemLLM drafts from text already in the prompt. StairCut sets the block size from the pack's price file, the measured cost of a verify step by block size. After a round with no draft, the engine takes 1, 2, 4, then 8 plain steps before it looks again.

A 4,096-token prompt reads at 5,045 tok/s. The weights take 28.5 GiB in memory and load in about 40 s.

Long-context cache (FP8), from TandemLLM 0.14.1 on

TandemLLM can keep the 14 full layers' keys and values as FP8 (e4m3) codes with one fp32 scale per head and token: 14,784 bytes a token instead of 28,672. It is off by default, because the output is then no longer the exact model's.

We measured it on this pack against the exact cache. The prompt was the first 122,880 tokens of one long source file, and we scored the last 1,024: KL 0.047, top-1 agreement 95.2 %, perplexity 4.097 against 4.062.

Plain decode in the engine, 64 greedy tokens at each point, one run each:

Reading a prompt gets 4 to 6 % slower at those lengths, since each chunk decodes the cached rows back to BF16.

To turn it on in the Setup Guide (the dashboard's #/setup), pick Flint, go to the Tools step and switch on "Long-context cache (FP8)" under Features. Then finish Review and Install. To turn it on from the command line, add this line at the end of ~/TandemLLM/run/ops/serve.env:

QWEN38_FLINT_KV_FP8=1

Then run ~/TandemLLM/bin/tandem restart. curl -s localhost:8000/health shows "kv_cache": "e4m3" under weights when it is on. To turn it off, switch it off in the Setup Guide, or delete the line and restart.

Tool calls: tool-eval-bench, from TandemLLM 0.14.1 on

TandemLLM 0.14.1 on one DGX Spark, 2026-10-22. tool-eval-bench 2.8.1.dev13 (commit f2e368b), seed 42, temperature 0, one request at a time, one run a row (n=1), a restart before each row. The server ran the Setup Guide's defaults for this pack: one slot, the shipped effort medium, the conductor on, the FP8 cache off. The raw rows send their own effort, thinking off or xhigh. The tools rows think at the server's medium, and the bench-pk rows at the level of each turn's class. Score, then the time the row took:

Raw rows send X-Tandem: raw, so the model answers alone; thinking off is reasoning_effort: none. The conductor rows pin a profile with X-Tandem-Profile: tools is what the conductor picks for a client that offers tools, and bench-pk is a test profile TandemLLM does not ship. Hard mode at n=1 varies a lot between runs, so we claim no ranking between the packs. The measurement file has every row with its misses.

How to run it

TandemLLM runs this pack from release 0.14.0 on. On a DGX Spark with TandemLLM 0.14.0 installed:

~/TandemLLM/run/ops/use-profile.sh flint-exl3-6.0

use-profile.sh downloads this repository at a pinned commit, checks each file against SHA256SUMS and starts the engine on port 8000 with the lookup drafting. Flint decides on each turn whether to reason in a <think> block; TandemLLM's effort levels cap that block, and thinking off answers at once.

These are EXL3 files with one architecture stock exllamav3 does not have: a loader needs the patch in convert/ (or its own Flint code) and must read expert_interm_div. We have not tested other EXL3 loaders.

How we made it

convert/exllamav3-1.6.2-flint.patch applies to exllamav3 v1.6.2 (commit 4c84273). It adds exllamav3/architecture/flint.py, a guard in the quantizer for all-zero weights, and an option in eval/model_diff.py to evaluate on a local text file. convert/README.md has the commands. We used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook, on one RTX PRO 6000. Before the conversion we checked the converter's forward against a plain fp32 implementation of the published model description: a tiny random Flint per layer within fp16 noise, and all 49 real layers at cosine 0.99998 or better. quality/ holds those checks and the version file.

License

Flint is released under Apache-2.0, and so are these weights. LICENSE is Recursive's, and NOTICE says where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed. Everything else about the model, including its intended use and its benchmarks, is in Recursive's model card.

Credits

Recursive made Flint and its BF16 checkpoint. turboderp made exllamav3 and the EXL3 format. TandemLLM is ours.