Flint-40B-A2.5B-abliterated (NVFP4 GGUF)
An abliterated (uncensored) Flint: RecursiveSI's Flint 40B-A2.5B with its refusal direction removed, for security research and coding work, as an NVFP4 GGUF for NVIDIA Blackwell GPUs with llama.cpp CUDA.
Summary: held-out MITRE ATT&CK refusals 15% → 0% (n=100, Q8_0). No significant drop on HumanEval+, MBPP+, GSM8K, MMLU-Pro, CyberMetric or WMDP-cyber (paired, same items). KL divergence from the original on harmless prompts: 0.0146. Builds: BF16, GGUF (Q8_0 to Q3_K_M), NVFP4 for NVIDIA Blackwell, MLX 4-bit for Apple silicon.
Which build
Not sure? Take the GGUF Q4_K_M (24 GB). It runs in llama.cpp and in apps built on it, including LM Studio (its llama.cpp runtime 2.63.0 includes Flint). On a Mac, the Rec 4-bit build runs in RecursiveSI's Rec runtime (125 tok/s decode on an M4 Pro) and the MLX 4-bit build in mlx-lm (85 tok/s).
| You run | Download | Size |
|---|---|---|
| llama.cpp, or an app that bundles llama.cpp b11827 or later; any GPU, CPU or Mac | GGUF: Q8_0 is the benchmarked build, Q4_K_M the usual choice |
42 / 24 GB |
| NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, DGX Spark) with llama.cpp CUDA | NVFP4 GGUF (this repo) | 23 GB |
| Apple silicon Mac with mlx-lm installed from GitHub (Flint support is merged but not released yet) | MLX 4-bit | 21 GB |
| Apple silicon Mac with RecursiveSI's Rec runtime | Rec 4-bit | 21 GB |
| transformers or SGLang on a GPU server; fine-tuning or making your own quants | BF16 safetensors | 80 GB |
Plan for the file size plus memory for context: about 26 KB per token, so roughly 7 GB at the full 262,144 tokens.
The file Flint-40B-A2.5B-abliterated-NVFP4.gguf (22.9 GB) uses the same tensor layout, types and scales as
RecursiveSI's flint-nvfp4-gguf, built from our BF16 weights.
Community builds
Quantizations made by other people. I have not tested them. The model tree on this page links every build that lists this model as its base.
| Build | By | For |
|---|---|---|
| MLX oQ4e (23 GB), MLX oQ8e (42 GB) | Studio-Research-M1 | oMLX on Apple silicon; the cards report KL divergence against FP32 and speed on an M1 Max |
| GGUF, imatrix GGUF | mradermacher | llama.cpp |
| FP8 | cbert33 | GPU servers (compressed-tensors W8A8, calibrated FP8 KV cache) |
| EXL3 4.0 bpw | cbert33 | ExLlamaV3 |
| REAP-224 Rec 4-bit (19 GB) | RinneCODER | Macs with less memory: the Rec build with 32 of the 256 routed experts removed in each layer |
Making your own quant: keep Flint out of FP16. In layer 46 the expert product silu(gate) · up reaches about 90,000, above the FP16 maximum of 65,504, so FP16 builds answer !!!! and give NaN logits. MLX computes in the dtype of the scales and unquantized tensors, so store those in BF16. Studio-Research-M1 found and measured this (discussion).
Results
Short paired check against RecursiveSI's NVFP4 file, both on llama-server (llama.cpp b11827) with the same flags, Flint's recommended sampling (temperature 1.0, top_p 0.95, top_k 20) and full reasoning, on an RTX PRO 6000 Blackwell. Same items for both models; a quality check fails only on a drop that is significant at 95%.
| Check | RecursiveSI NVFP4 | Abliterated NVFP4 | Result |
|---|---|---|---|
| MITRE refusals (n=50), held out from the search | 22% | 0% | pass |
| MITRE replies with no final answer (still reasoning at 8,192 tokens) | 2 | 12 | info |
| GSM8K (n=40) not significantly worse | 95% | 95% | pass |
| HumanEval+ (n=40) not significantly worse | 90% | 92.5% | pass |
| pp512 tok/s within 3% | 6814 | 6815 | pass |
| tg128 tok/s within 3% | 207.4 | 207.5 | pass |
Heretic's default refusal count, measured on the NVFP4 builds (100 mlabonne/harmful_behaviors test prompts, reasoning off, greedy, 100 tokens): original 96/100, abliterated 55/100. That count flags any reply containing words such as "harmful" or "illegal", including answers that comply; only 1 of the 55 replies opens with a refusal. Meta's CyberSecEval refusal check on the same replies: original 18/100, abliterated 1/100.
The full benchmark set (Q8_0, more items) is on the main card.
Usage
Use a llama.cpp CUDA build with Flint support (b11827 or later). Plan for at least 48 GB of GPU memory, or unified memory on DGX Spark.
hf download dima6312/Flint-40B-A2.5B-abliterated-NVFP4-GGUF Flint-40B-A2.5B-abliterated-NVFP4.gguf --local-dir .
llama-server -m Flint-40B-A2.5B-abliterated-NVFP4.gguf \
-ngl all -c 262144 -fa on --jinja \
--temp 1 --top-p 0.95 --top-k 20 --min-p 0
Reasoning is optional; when the model reasons, it uses <think>…</think> before its answer. Give long security tasks
a large token budget: on the main checks, 19 of 100 MITRE answers were still reasoning at 8,192 tokens.
Limitations
- The check above is short (n in each row). Refusals are lower, not zero.
- Abliteration removes refusals; it adds no knowledge. Mistakes the original makes, this model makes too.
Responsible use
This model answers requests the original refuses. It is meant for authorized security testing, research, education and defensive work. You are responsible for how you use it and for following the law and the terms of the systems you test. Its answers can be wrong; check them before acting on them.
Credits
- Flint and the NVFP4 GGUF layout by RecursiveSI (Apache 2.0). This is a derivative of their weights under the same license.
- abliterix by wuwangzhang1216, building on Heretic by p-e-w.
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
- Meta's CyberSecEval; GSM8K (OpenAI); EvalPlus.
- llama.cpp (ggml-org).