Flint-GGUF-Serve-mixed-4-8bit
A llama.cpp GGUF of Recursive Flint (39.8B total / 2.5B active, coding-agent MoE), sized so that the weights plus a 128K f16 KV cache fit on a 32 GB GPU with about 3 GB to spare. Built and measured on an AMD Radeon AI PRO R9700 (32 GB, Vulkan).
It follows the same recipe as Flint-MLX-Serve-mixed-4-8bit: routed experts are low-bit, and everything that runs for every token is 8-bit.
File size: 23.28 GiB (25.0 GB), 5.03 BPW. Recommended VRAM: 32 GB for the full 131K context with no spill to system memory.
Recipe
Quantized with llama-quantize (llama.cpp b11836) from ggml-org/Flint-GGUF BF16, using unsloth's imatrix.
| Tensors | Type |
|---|---|
Routed experts gate/up (ffn_{gate,up}_exps) |
Q4_K |
Routed experts down (ffn_down_exps) |
Q5_K |
| Attention q/k/v/o, shared expert, dense MLP (layers 0–1) | Q8_0 |
| Token embeddings, LM head | Q8_0 |
Router (ffn_gate_inp), norms, attention sinks |
F32 (unquantized) |
Command (tensor-types.txt below):
llama-quantize --imatrix imatrix_unsloth.gguf --tensor-type-file tensor-types.txt \
--token-embedding-type q8_0 --output-tensor-type q8_0 Flint-BF16.gguf Flint-GGUF-Serve-mixed-4-8bit.gguf Q4_K_M
ffn_(gate|up)_exps=q4_k
ffn_down_exps=q5_k
attn_(q|k|v|output)=q8_0
ffn_(gate|up|down)_shexp=q8_0
blk\.[01]\.ffn_(gate|up|down)=q8_0
ffn_gate_inp=bf16
Why
unsloth's UD-Q5_K_XL is 27.9 GiB. On a 32 GB card at 128K context it spills about 2 GB into shared system memory, and in real agent sessions generation then drops from about 120 to about 60 tok/s. This build stays fully resident.
llama-server
llama-server -m Flint-GGUF-Serve-mixed-4-8bit.gguf -ngl 99 -fa on -c 131072 -ub 512 --jinja \
--temp 1.0 --top-p 0.95 --top-k 20
Results (R9700, Vulkan, llama.cpp b11836)
Weights 23.28 GiB (5.03 BPW). With the full 131,072-token f16 KV cache and -ub 512, the GPU uses 28.1 GiB dedicated with nothing spilled to shared memory (Windows GPU counters, desktop included). That leaves about 3.7 GiB free.
llama-bench -fa 1 -p 2048 -n 128 -d 0,32768,131072, tok/s:
| ubatch | depth | prefill (pp2048) | decode (tg128) |
|---|---|---|---|
| 512 | 0 | 3,859 | 122.0 |
| 512 | 32K | 2,314 | 100.7 |
| 512 | 131K | 1,013 | 68.7 |
| 2048 | 0 | 5,553 | 121.2 |
| 2048 | 32K | 2,779 | 101.2 |
| 2048 | 131K | 1,116 | 68.9 |
-ub 2048 adds about 0.66 GiB and starts to touch shared memory at 131K on a 32 GB card. For full context, use 512.
Long-context retrieval: a 93,574-token prompt (repository source and docs, with a passphrase planted in the middle) through an OpenAI-compatible router. Prefill ran at 1,915 tok/s (48.9 s), decode at 76.8 tok/s, and both answers were correct.
Agentic use: it runs the pi coding agent through OmniRoute with prompt-cache reuse. At a 40K+ context, turns take 1–4 s and decode runs at 92–105 tok/s.
For comparison, unsloth UD-Q5_K_XL (27.9 GiB) on the same card at 131K spills about 2 GB to system memory, and real agent sessions fall to about 60 tok/s decode and about 870 tok/s prefill.