Flint 40B-A2.5B, EXL3

Flint is Recursive's mixture-of-experts model for coding agents. This is a 3.43 bits-per-weight EXL3 quantisation of RecursiveSI/flint, made by Yamz Labs. It runs on one 24 GB NVIDIA card with the Kyojin engine, up to the full 262,144 tokens of context.

Model size: 39.8B parameters, about 2.5B active per token. The pack is 17.9 GB (16.7 GiB) in 3 shards. The parameter count in the Hugging Face sidebar is not a model figure: the site counts the packed 16-bit words of the EXL3 format.

Hardware: tested on one NVIDIA RTX 3090 (24 GB) with the Kyojin engine. The pack is in the standard EXL3 format; other cards and other runtimes are untested.

Highlights

• Fits a 24 GB card, with the whole context. Weights take 16.7 GiB. With a 6-bit KV cache the server holds 262,144 tokens at a peak of 21.0 GiB. With a 16-bit KV cache it holds about 131K.
• Fast on one RTX 3090. Decode 133.5 tok/s at 4K of context, 122.9 at 32K, 102.6 at 128K and 81.3 at 256K. Prefill (reading the prompt) 6282 tok/s at 4K, 2995 at 128K and 1807 at 256K.
• Quality. Against an fp32 run of the original model, KL is 0.125 on raw text and 0.0547 on text the model sampled itself, with top-1 agreement of 88.4 % and 93.3 % (measured on stable positions only, see Quality). At the same size this is 23 % and 34 % lower KL than our earlier plain conversion.
• Loops. No loop in 105 greedy generations.
• Agent work. 32 of 32 multi-turn agent tasks solved, with shell and file tools, 5.5 model turns per task. Flint thinks longer than a dense 27B model, so a task takes more wall time even at 3 times the decode speed (see Evals).
• Tool calls and thinking. The Kyojin server parses Flint's <think> blocks into reasoning_content and its tool calls into OpenAI tool_calls.
• Feedback wanted. Run it on your own card and tell us your tok/s: open an issue at https://github.com/Yamz-Labs/kyojin/issues or start a discussion on this page.

How it was made

This pack is our own conversion. It is not a copy of anyone else's quantisation. We ran the EXL3 conversion of Kyojin (the Yamz engine built on ExLlamaV3) on the BF16 checkpoint of RecursiveSI/flint:

• Bit widths per tensor. Attention, shared experts and the two dense layers: 6 bit. Routed experts: 3 to 6 bit, chosen by depth from a measured sensitivity of each layer and tensor class. Layers 46 and 48: gate and up projections at 5 to 6 bit. Output head: 6 bit. The router and the embeddings stay in fp16. Average: 3.43 bits per weight.
• Calibration. 250 rows of 2048 tokens: 70 % text sampled from the model itself, 30 % raw text rows. The Hessian of each routed expert is weighted by how often the router picks that expert on this data.
• Size. The recipe was fitted to the byte budget of a plain 3.4 bpw conversion, so the comparison with it below is at equal bytes.

The conversion tooling is not published. The engine and these weights are.

Run

Kyojin 1.7.0 or later. Build the NVIDIA extension (CUDA 13.0, torch 2.10 cu130, gcc 13, details in tools/cuda/README.md):

git clone https://github.com/Yamz-Labs/kyojin && cd kyojin
tools/cuda/build.sh /path/to/cuda13 /path/to/venv-cuda/bin/python
python tools/flint/serve.py --model /path/to/this-pack --port 8000 -c 262144 --kv 6,6

OpenAI-style HTTP API (/v1/chat/completions, streaming and not, /v1/models, /health). Flags:

• -c context in tokens (default 131072). --kv K,V bits of the KV cache for the full-attention layers: 16, 8,8, 6,6, 4,4. On NVIDIA the default is 6,6. For 16-bit KV use --kv 16 -c 131072.
• --max-batch parallel requests (default 4). --default-max-new reply budget when a request has no max_tokens (default 32768, thinking included).
• --ngram N optional n-gram drafting, minimum match length N (default off). It gave +9 % decode on code prompts and does not change the answers.

The server reads the chat template and the sampling values from this folder. There is no grammar constraint on tool calls (the vendor server has one).

Hardware

Peak GPU memory in the speed runs (6-bit KV cache): 17.8 GiB at 4K of context, 21.0 GiB at 256K. A 16-bit cache peaks at 21.4 GiB at 128K and does not fit 256K. An 8-bit cache fits 256K at 21.9 GiB, with no room to spare.

Quality (reference = fp32 run of the original model)

The reference is Flint run in fp32 by our engine from the original BF16 weights. KL measures how far the output distribution is from the reference (lower is closer). Top-1 is how often the most likely token is the same.

The held-out rows are not in the calibration set. The calibration mix is 70 % self-sampled text, so the second row is closer to the calibration domain than the first. Treat the raw-text row as the fair figure.

Positions left out. The BF16 reference itself is numerically unstable at some positions: re-running it in another batch, or in fp32, gives a different distribution (KL above 0.3). It happens on repetitive or copy-heavy text, where several attention matches are nearly tied. We leave out every position where our fp32 and BF16 references disagree by KL above 0.3. That is 23 % of the code positions, 1 to 9 % of the other raw domains and 0.2 to 1 % of the self-sampled rows. All figures above are for the remaining, stable positions. Figures with those positions included are higher, for every model, including the original one.

KV cache quality

Teacher-forced, 512 tokens after 32K of real text, against the same model with a 16-bit cache:

On 2K-token documents, the KL against the fp32 reference is 0.0547 with a 16-bit cache, 0.0542 with 8/8, 0.0594 with 6/6 (+9 %) and 0.0732 with 4/4. We recommend 6/6 for long context and do not recommend 4/4. A needle-in-a-haystack test at 262,144 tokens with the 6/6 cache found the needle 3 times out of 3.

Evals

Same card and same sampling (temperature 1.0, top-p 0.95, top-k 20, each model's chat template, 3072 new tokens at most, one sample per problem). This pack and Qwen3.8-27B ran on Kyojin; MiniCPM-V ran on llama.cpp, in an earlier session. The sets are small: one problem more or less is noise.

On this small bench MiniCPM-V 4.7 leads on code and tools. Flint's gains are speed and the room it leaves for long context.

Agent run: 32 multi-turn tasks (bug fix, CSV work, rename across files, shell, log reading, spec to code, two-bug fix, small command-line tools), an open agent harness with shell and file tools, 32K-token reply budget, same harness and sampling for both models.

The tasks are too easy to separate the two on quality. Flint decodes 3 times faster but writes about 3 times more tokens per task, so a task takes longer.

Speed

One user, greedy, real text, one fresh process per depth, 64 new tokens, CUDA graph on, tokens per second. The depth is the prompt length.

Decode at 256K is 61 % of decode at 4K. The 6/6 cache decodes as fast as the 16-bit cache at 4K and faster at every depth beyond, because it moves less data. A 262,144-token prompt takes about 156 s to the first token.

Against llama.cpp on the same card

llama.cpp master (commit 970b51c, CUDA 13, sm_86) with the community GGUF UD-Q3_K_XL (17.2 GiB), the same prompts, greedy, decode | prefill in tok/s:

Time to the first token on a 32K prompt: 7.4 s with Kyojin, 7.7 to 9.0 s with llama.cpp.

Read this table with three cautions. The rivals ran in one measurement session against an older Kyojin build; the Kyojin rows here are lower than in the speed table above, and the later prefill work was not re-run against llama.cpp. We claim no prefill lead: in this session llama.cpp prefilled faster than Kyojin at 4K (by 8 to 10 %), and Kyojin was ahead at 32K and beyond. The weights differ (EXL3 against GGUF), so quality is not compared. Kyojin decodes faster at every depth in this table.

MiniCPM-V 4.7 35B-A3B Q4_K_M on llama.cpp (Vulkan, flash attention, no draft model) decoded at 145 tok/s at 4K and 128 at 32K, and prefilled at 3242 and 2763. Qwen3.8-27B (EXL3 4.0 bpw) decodes without speculation at 45.6, 41.5, 37.2 and 32.8 tok/s at 4K, 32K, 128K and 256K.

Sampling

Use the base-model values: temperature 1.0, top-p 0.95, top-k 20. For hard problems allow an output budget of at least 32768 tokens, thinking included: Flint thinks for a long time and a short budget cuts the answer. The Kyojin server applies these values when a request leaves them out.

Limits

• Tool calls have no grammar constraint in the Kyojin server. A malformed call is possible; check it before you run it.
• With several users at once, an answer can differ from the same request run alone, at positions where two tokens are nearly tied. Chunk shape also moves the output slightly (KL noise floor 0.011 to 0.028).
• The quality figures skip the positions where the reference is unstable (see Quality). Do not read them as figures over all text.
• The 6/6 KV cache is lossy (+9 % KL on short documents). Use --kv 16 up to 131K if you want none.
• The 3090 runs power-capped; speed on another card or power limit differs.
• Only one resident model and one GPU are supported by tools/flint/serve.py.
• Flint thinks longer than a dense model of similar quality. Plan for the wall time.
• This is not a quantisation validated by the model's authors. Test generated code and review consequential changes before you apply them, as the base model card says.
• The conversion tooling is not published.

Licence and credits

Apache 2.0. Base model: Flint by Recursive, RecursiveSI/flint, Apache 2.0. The licence file is included as LICENSE. Modification: the weights were quantised and converted to the EXL3 format by Yamz Labs; the architecture, tokenizer and chat template are unchanged. Our additions: Copyright (c) 2026 Yamz Labs, Apache 2.0. Provided as is, without warranty.

Base model facts, from the upstream card: 39.8B parameters in total and about 2.5B active per token; 49 layers (2 dense, then 47 mixture-of-experts layers); 35 sliding-window layers (513 tokens) and 14 full-attention layers without positional encoding; 32 query heads and 4 key/value heads of dimension 128; 256 routed experts per layer, 6 selected per token, plus an always-active shared expert; vocabulary of 150,144 tokens; 262,144 tokens of context. The upstream checkpoint is about 80 GB and is served with a vendor kit on data-centre GPUs.

Engine: Kyojin, built on ExLlamaV3 by turboderp (MIT). Quantisation, recipe, calibration and packaging by Yamz Labs. This release is unofficial. Not affiliated with Recursive or turboderp. "Flint" is a name of Recursive.

Training data of the base model: see the upstream card. Our calibration used 250 rows of 2048 tokens, 70 % text sampled from the model itself and 30 % raw text.