Flint-abliterated GGUF

The uncensored Flint coding-agent model on one 16 GB or 32 GB GPU.

12 GB and 24 GB builds · on par with unsloth's UD files of the same size · llama.cpp

OrcaRouter AI Gateway · X · Discord · GitHub · All Models

---

Abliterated Flint, run locally.

These are GGUF builds of an abliterated (refusal-removed) RecursiveSI/flint. Flint is a 39.8B MoE for coding agents, with 256 routed experts (top-6) and one shared expert, about 2.5B active per token. It uses hybrid sliding-window / full attention with learned attention sinks, and a 262K-token context.

One refusal direction was removed from the BF16 weights before quantization. Harmful-prompt refusal drops from 86% to about 10%, and general capability stays within noise.

Routed experts are 95% of the weights, so that is where the bits were spent. Each group of expert layers got its own precision from measured KL divergence. Every tensor was then rounded with GPTQ-style error compensation into stock llama.cpp types, so no custom kernels are needed.

At about the same file size, the builds match unsloth's UD files on held-out text the search never saw. Mix-24G scores slightly lower KLD than UD-Q4_K_XL while being 1.1 GB smaller, and Mix-12G sits between UD-IQ1_M and UD-IQ2_XXS.

---

⚠️ Disclaimer: read before use

This model is abliterated: its safety alignment has been substantially reduced. It will comply with many harmful, unethical, offensive, or illegal requests that the original Flint would refuse.

• It is released strictly for legitimate research: interpretability, AI safety and refusal-mechanism study, red teaming, and robustness evaluation.
• You assume full responsibility for how you use it and for everything it generates. Add your own safety, moderation and abuse-prevention layers before any deployment.
• Use must comply with the Apache 2.0 license inherited from the base model, and with all applicable law.
• The authors accept no liability for misuse. Its outputs do not reflect the views of the uploaders or of Recursive.

---

Pick a build

• Mix-24G fits a 32 GB GPU with room for a long context. It is close to the BF16: 96.5% of next tokens match, and perplexity is 1.1% higher.
• Mix-12G fits a 16 GB GPU. It averages 2.3 bits per expert weight, and that costs quality: 88% of next tokens match, and perplexity is 20% higher.

Use Mix-24G if you have the memory. Use Mix-12G when 16 GB is what you have.

The KV cache is small. Only the 14 full-attention layers keep the whole context, at about 28 KB per token in F16, and the 35 sliding-window layers keep 513 tokens each. So 64K tokens of context take about 1.9 GB, and 128K tokens take about 3.7 GB.

---

Evaluation

Fidelity to the BF16

All builds are scored against the BF16 GGUF they were quantized from, using llama.cpp's full-vocabulary KLD.

The text is Flint's own answers. Prompts come from four sources:

• code (Magicoder OSS-Instruct), 35%
• English chat (UltraChat 200k), 25%
• math reasoning (OpenR1-Math), 25%
• non-English chat (WildChat), 15%

The abliterated BF16 answered each one with Flint's recommended sampling (temperature 1.0, top_p 0.95, top_k 20). Each conversation was rendered through the chat template with its reasoning as a <think> block. KLD is counted on the assistant tokens, reasoning and answer, over 128 windows of 2048 tokens. The precision search used a separate draw and never saw this text.

Each build is compared with unsloth's dynamic GGUFs around the same size, from unsloth/flint-GGUF. Unsloth quantized the original Flint, so their files are scored against the original BF16, with the same text and llama.cpp build. Each KLD therefore measures the quantization error alone.

• Mix-24G matches UD-Q4_K_XL in mean KLD and beats it in next-token agreement, in a file 1.1 GB smaller. Its tail (KLD p99) is 5% higher.
• Mix-12G has 8% less KLD than UD-IQ1_M and 3% more than UD-IQ2_XXS, with the highest next-token agreement of the three. Its tail is the weakest of the three.

Why the model's own text. Flint always reasons before it answers. On other models' answers (no reasoning block, someone else's style) a model makes confident misses even at high precision, and which of those misses a quant happens to flip dominates the KLD. So the score covers what the model itself writes.

Teachers. The abliterated BF16 is itself 0.015 KLD from the original on this text. Scored against the abliterated BF16 instead of its own, UD-IQ2_XXS reads 0.2005, so the comparison does not hinge on the choice of teacher.

Method note. These are full-vocabulary KLD and next-token agreement at 2K context, not a downstream benchmark.

Abliteration

These numbers were measured on the abliterated BF16 before quantization, against the original Flint, with identical prompts, splits and seed. Generation used Flint's official SGLang kit and sampling (temperature 1.0, top-p 0.95, top-k 20). Refusal is judged on the final answer by a rule-based classifier, not an LLM judge, so treat them as indicative.

Harmful-prompt refusal (lower = more uncensored). Average 10.4%, against 86.3% for the original.

Over-refusal on benign prompts (lower = better).

Capability. Every delta is within ±2 pp, which is sampling noise at these sizes.

---

How the builds were made

Where the bits go

The budget is the whole file: 12.0 GB and 24.0 GB.

1. Measure. For every group of tensors, the search measures the real KLD of moving it one precision level up and one level down. The 85 groups are:

   • routed experts: one projection (gate, up or down) over about two layers, 25 groups each
   • attention: q, k, v and output, one group each over all layers
   • shared expert: gate, up and down, one group each
   • dense FFN: layers 0–1
   • embedding and output head: one group each

   Each measurement runs llama.cpp's own kernels on 64 windows of 2048 tokens of the model's own text.

2. Choose. A knapsack picks the combination of moves that fits the budget exactly.

3. Accept. Only steps whose candidate KLD was actually measured lower are kept.

Mix-12G puts its bits as follows:

Mix-24G puts its bits as follows:

In both builds the router, the norms and the attention sinks stay F32.

How each tensor is rounded

llama-quantize rounds each block of 256 weights on its own, weighted by the importance matrix. Rebuilt that way from the same BF16, with the same per-tensor types, unsloth's UD-IQ2_XXS scored 0.27 KLD against 0.196 for their own file, with either importance matrix. The gap was in the rounding, not in the allocation or the importance matrix.

So each tensor is rounded the GPTQ way instead. It is quantized one GGUF block of columns at a time, with llama.cpp's own quantizer for that type. Each block's rounding error is then pushed onto the columns not yet quantized, through the inverse Hessian of the layer's inputs, so later columns absorb what earlier ones lost. The output keeps stock types and layout, so any llama.cpp build reads it.

• The Hessians come from 424K tokens of Flint's own conversations, run through the BF16 model.
• A routed expert sees only about 2.3% of those tokens, so its Hessian is shrunk toward its layer's mean, with a prior of 2048 tokens. The prior and the damping (0.3) were tuned on held-out KLD.

With this rounding and the search rerun on top of it, the 12 GB build went from 0.262 to 0.202 KLD, and the 24 GB build from 0.0222 to 0.0186.

The importance matrix comes from ~1.2M tokens:

• the exllamav3 calibration mix: web text (C4), code, multilingual text, technical text, Wikipedia, tiny stories and random token rows
• Flint's own conversations, so that it sees the chat format and <think> blocks

None of it overlaps the evaluation text.

---

What's in this repo

Each build is a single file with the chat template embedded. These are text-only, like Flint.

---

Serving (llama.cpp)

Flint support landed in llama.cpp master with ggml-org/llama.cpp#30928, and its chat template needs #30929. These files were made and tested at master commit e0f142d:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout e0f142d   # or any later master
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli
# Mac: cmake -B build (Metal is on by default) · CPU only: cmake -B build

hf download orcarouter/Flint-abliterated-GGUF Flint-abliterated-Mix-24G.gguf --local-dir ./flint-gguf

# one 32 GB GPU (Mix-24G), 128K context
llama-server -m flint-gguf/Flint-abliterated-Mix-24G.gguf -c 131072 --jinja

# one 16 GB GPU (Mix-12G), 32K context
llama-server -m flint-gguf/Flint-abliterated-Mix-12G.gguf -c 32768 --jinja

• Reasoning is always on. With --jinja, llama-server returns the reasoning in message.reasoning_content and the answer in message.content. Give it a large max_tokens: Flint's own card recommends at least 32768 for hard problems, reasoning included.
• Tool calls work through the OpenAI-compatible API with --jinja, and come back in message.tool_calls.
• Sampling. Recursive recommends temperature=1.0, top_p=0.95, top_k=20. The files carry these as their default sampling metadata.
• Mac. macOS lets the GPU wire only about 75% of RAM by default. Mix-24G on a 32 GB Mac needs the limit raised, for example sudo sysctl iogpu.wired_limit_mb=28000, and other apps closed.

---

Model details

---

Bias, risks, and limitations

• Safety guardrails are substantially reduced. See the disclaimer.
• Mix-12G is noticeably degraded. About 2.3 bits per expert weight is a large cut. It is the best we found at this size, not a match for the BF16.
• Refusal and capability were measured on the BF16, not on these files. The quantized builds were checked for fidelity (KLD) only.
• Not measured: coding and agentic benchmarks (Flint's card reports Terminal-Bench and SWE-bench; none of that was rerun here), and long context. All fidelity measurements are at 2K context.
• Tested on CUDA only. CPU and Apple Metal have not been run with these files.
• Inherits the biases and limitations of RecursiveSI/flint.
• Requires llama.cpp with PR #30928 and #30929. Tools built on older llama.cpp, such as older Ollama and LM Studio releases, will not load these files.

License

Apache 2.0, inherited from the base model RecursiveSI/flint. Abliteration and quantization do not change the license obligations.

---

One Gateway. Every Model.

Route Smarter · Ship Safer · Spend Less

Website · Models · GitHub · Discord · X