Flint — ROCmFPX GGUF

Update 2026-10-24: much better 4-bit files. Please download again. If you downloaded flint-Q4_0_ROCMFP4_STRIX.gguf or flint-Q4_0_ROCMFP4_FAST.gguf, replace it with one of the new 4-bit files below. They are about three times closer to the original model (KLD 0.18 instead of 0.54) and now beat a stock Q4_K_M while being smaller. The 8-bit and 6-bit files did not change.

RecursiveSI/flint, Recursive's 39.8B mixture-of-experts model for coding agents (about 2.5B active per token), quantized to the ROCmFPX formats, a llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

Start with Q4_0_ROCMFP4_Q8ATTN (23.3 GB). It is closer to BF16 than a stock Q4_K_M of the same model while being smaller. Use Q6_0_ROCMFPX or Q8_0_ROCMFPX if you have the memory.

These files need HarshaVardhan3002/ROCmFPX at commit b130d4995 or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. These files have not been tested on AMD hardware yet.

Not an official release of Recursive. All credit for the model goes to its authors.

Files and measured quality

Measured against the BF16 model (perplexity 19.1433) on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the next-token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16.

The 6-bit file is named flint-Q6_K_ROCMFPX.gguf so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock llama.cpp Q6_K, and it needs the same fork as the other files.

For comparison, a stock llama.cpp Q4_K_M of the same model (23 GB), scored the same way, measured PPL 19.92, KLD 0.2449, same top 82.94%. Both 4-bit files here are closer to BF16 and smaller.

The 4-bit rows were measured in a second run with a fresh BF16 reference (perplexity 19.2141; the same method gives KLD 0.0284 for the BF16 model itself, which is the floor of this measurement). The 8-bit and 6-bit rows come from the first run.

The 4-bit recipe

Flint has 256 small experts per layer, and they hold about 95% of the weights. The rest (attention, the shared expert, the two dense layers at the start, and the output layer) is only about 2B weights, but measurements showed that putting it at 4-bit caused most of the loss: with the experts kept at BF16 and everything else at 4-bit, KLD was already 0.473. So both 4-bit files keep all of that at Q8_0_ROCMFPX (token embedding Q6_K) and only the routed experts are 4-bit: dual-scale Q4_0_ROCMFP4 in the recommended file, single-scale Q4_0_ROCMFP4_FAST in the speed file. This costs about 1 GB over an all-4-bit file.

These replace the first 4-bit files (2026-10-24): Q4_0_ROCMFP4_STRIX (21.3 GB, KLD 0.5360) and Q4_0_ROCMFP4_FAST (21.2 GB, KLD 0.5428), which also kept attention and the output layer at 4-bit.

Not published: Q3_0_ROCMFPX (22.1 GB, KLD 0.5523) is far behind the 4-bit files; Q2_0_ROCMFPX does not work on this model.

Why these files need a newer build

Flint's weights are about 20 times smaller than those of most models. ROCmFPX stores one small scale per block of 32 weights, and that scale has a fixed smallest step, so on Flint many blocks lost most of their precision: a first 8-bit file had KLD 3.9 (same top 33%) (ROCmFPX#118).

These files were made with an opt-in fix in llama-quantize (LLAMA_ROCMFPX_TENSOR_SCALE=1): each weight tensor (each expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing, and the inverse factor is stored as a small extra .scale tensor that llama.cpp already applies to the matmul output. The factor is a power of two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject.

How these were made

1. convert_hf_to_gguf.py --outtype bf16.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (calibration_datav3), 512-token chunks.
3. LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix, one run per type, from HarshaVardhan3002/ROCmFPX b130d4995. The 4-bit files use Q4_0_ROCMFP4_STRIX with --tensor-type "attn_(q|k|v|output)=q8_0_rocmfpx" --tensor-type "_shexp=q8_0_rocmfpx" --tensor-type "blk\.[01]\.ffn_(gate|up|down)\.=q8_0_rocmfpx" --output-tensor-type q8_0_rocmfpx, plus --tensor-type "_exps=q4_0_rocmfp4" for the recommended file.
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels. Each quant was expanded back to F16 (llama-quantize --allow-requantize ... F16), which keeps its stored weights and scales, and scored on the same GPU with stock F16 kernels. The fork's own CUDA kernels for these types are not correct on NVIDIA, so they were not used. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m flint-Q4_0_ROCMFP4_Q8ATTN.gguf -ngl 999 -fa on --jinja --temp 1.0 --top-p 0.95 --top-k 20

Sampling settings are the ones Recursive used for its evaluations.

License

Apache-2.0, same as the base model.