GGUF builds of Gemma-4-12B-Uncensored for llama.cpp, Ollama and LM Studio: seven sizes from IQ3_M to Q8_0, plus the image projector and Google's assistant draft model for faster generation. The model is Gemma 4 12B-it with its refusals trained out by Abacus.AI. On 241 harmful prompts from HarmBench and JailbreakBench, stock Gemma 4 12B-it refuses 91.7% with thinking on; the Q4_K_M file here refuses 4.1% (10 of 241), measured on the file itself in llama.cpp and the same as the full-precision weights. With thinking off the figures are 95.4% and 16.6% (40 of 241). Every file was loaded and tested with thinking on, thinking off and an image, and the default download was run in llama.cpp, Ollama and LM Studio.
Built with llama.cpp b8abe41 from the BF16 weights. IQ3_M, Q3_K_M, IQ4_XS and Q4_K_M use an importance matrix (imatrix.gguf, included) computed on 48K tokens of prose, code, dialogue, math, science and multilingual text. Training, the full refusal evaluation and the known limitations are on the source card.
Share of 241 harmful prompts refused or hedged, scored by an LLM judge on the full answer. Lower means the model answered. The Q4_K_M bar is the file in this repository run in llama.cpp; the other two bars are full-precision weights.
| Quant | File | GPU memory8K context | GPU memory32K context | Pick it when you have |
|---|---|---|---|---|
IQ3_M | 5.73 GB | 7.0 GB | 7.4 GB | Smallest file. 8 GB of GPU memory. Scores below the 4-bit files (section 4). |
Q3_K_M | 6.09 GB | 7.3 GB | 7.7 GB | 8 GB of GPU memory. Scored no higher than the smaller IQ3_M (section 4). |
IQ4_XS | 6.64 GB | 7.9 GB | 8.3 GB | 10 GB of GPU memory; on an 8 GB card a few layers go to the CPU. |
Q4_K_M | 7.38 GB | 8.6 GB | 9.0 GB | The default (llama.cpp and Ollama pick it). 12 GB of GPU memory. |
Q5_K_M | 8.55 GB | 9.6 GB | 10.0 GB | 12 GB with a shorter context. |
Q6_K | 9.79 GB | 10.8 GB | 11.2 GB | 16 GB, or a 12 GB card with a short context. |
Q8_0 | 12.67 GB | 13.5 GB | 13.9 GB | Close to full precision. 16 GB and up. |
llama.cpp (downloads the image projector too):
llama-server -hf abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M --jinja -ngl all -c 8192 \
--temp 1.0 --top-p 0.95 --top-k 64
The model thinks by default, and thinking on is the mode it was tuned for. To turn thinking off, add --chat-template-kwargs '{"enable_thinking": false}', or send "chat_template_kwargs": {"enable_thinking": false} with a request.
Faster generation with the draft model (llama.cpp): download the draft file and add it to the command.
hf download abacusai/Gemma-4-12B-Uncensored-GGUF mtp-gemma-4-12B-it-Q8_0.gguf --local-dir .
llama-server -hf abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M --jinja -ngl all -c 8192 \
--temp 1.0 --top-p 0.95 --top-k 64 \
-md mtp-gemma-4-12B-it-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 -ngld all
Ollama:
ollama run hf.co/abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M
Thinking is on by default and --think=false turns it off. Ollama 0.40.16 rejects an explicit "think": true for Gemma 4 GGUF files pulled from the Hub, so leave it unset to keep thinking on.
LM Studio: search for abacusai/Gemma-4-12B-Uncensored-GGUF and pick a file from the table above.
Recommended sampling is the base model's: temperature 1.0, top_p 0.95, top_k 64.
Quantising to Q4_K_M did not shift refusals with thinking on: the file and the full-precision weights each refuse 10 of the 241 harmful prompts (paired exact test, p 1.0). With thinking off the file refuses 40 against 31; 16 prompts are refused only by Q4_K_M and 7 only by the full-precision weights (p 0.09), a gap this prompt set cannot separate from sampling noise. Stock Gemma 4 12B-it refuses 221 and 230.
| Q4_K_Mllama.cpp | Full precisionvLLM | Gemma 4 12B-itbase, full precision, vLLM | |
|---|---|---|---|
| Harmful prompts refused, thinking onthe default; lower is better | 4.110/241 | 4.110/241 | 91.7221/241 |
| Harmful prompts refused, thinking offlower is better | 16.640/241 | 12.931/241 | 95.4230/241 |
| BF16full precision | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | IQ4_XS | Q3_K_M | IQ3_M | |
|---|---|---|---|---|---|---|---|---|
| MMLU-Pro subset280 questions, exact match | 75.4211/280 | 75.0210/280 | 76.8215/280 | 73.6206/280 | 73.6206/280 | 70.7198/280 | 63.9179/280 | 65.4183/280 |
| IFEval541 prompts, prompt-level loose | 95.6517/541 | 95.7518/541 | 95.7518/541 | 95.4516/541 | 95.0514/541 | 93.3505/541 | 90.4489/541 | 93.7507/541 |
| Answers cut off at the capof 821; lower is better | 1.08/821 | 0.54/821 | 1.08/821 | 1.311/821 | 1.210/821 | 3.730/821 | 8.267/821 | 5.142/821 |
These files track their own full-precision weights more closely than stock Gemma 4 12B-it quantized the same way tracks its own, at every size. The absolute values are far higher than for most model families because Gemma 4 handles this raw-text test poorly even in full precision, so read the table as a comparison with stock and use the task scores above to choose a file.
| Quant | Mean KLDthis repository | Mean KLDstock Gemma 4 12B-it | Same top tokenthis repository | Same top tokenstock Gemma 4 12B-it |
|---|---|---|---|---|
IQ3_M | 1.683 | 2.053 | 53.0% | 46.2% |
Q3_K_M | 1.334 | 1.669 | 60.5% | 52.5% |
IQ4_XS | 0.886 | 1.209 | 69.5% | 59.8% |
Q4_K_M | 0.624 | 0.795 | 73.3% | 67.2% |
Q5_K_M | 0.669 | 0.972 | 73.6% | 64.0% |
Q6_K | 0.366 | 0.528 | 81.9% | 74.1% |
Q8_0 | 0.193 | 0.285 | 87.9% | 83.9% |
llama-perplexity --kl-divergence; lower is closer. "Same top token" is how often the quant's most likely next token equals BF16's. The stock columns are google/gemma-4-12B-it converted and quantized here with the same converter, settings and calibration text, each file against its own BF16. Full-precision perplexity on this text is 595 for stock Gemma 4 12B-it and 692 for this model. Stock builds by others, measured the same way against the stock BF16: the ggml-org Q8_0 gives exactly the stock Q8_0 values above, and the bartowski Q4_K_M 0.837 (66.0%). Bold = closer of the pair.
mtp-gemma-4-12B-it-Q8_0.gguf is Google's assistant draft model for Gemma 4 12B-it, converted unchanged; it was not retrained for this model and still drafts well for it. With Q4_K_M and 3 draft tokens, 61.9% of drafted tokens were accepted and generation ran at 228 tokens per second against 119 without the draft (1.91x), on 24 prompts (12 MMLU-Pro, 12 IFEval) with thinking on and up to 1,024 tokens each, one request at a time on an RTX PRO 6000.
llama.cpp renders this repo's embedded chat template exactly as the Hugging Face tokenizer does (checked for a thinking-on turn, a thinking-off turn and a tool-call round trip), so the thinking switch and the tool-call format are the same as in the full-precision release.
Refusals were measured on Q4_K_M only. Audio and video input, the full 262,144-token context, tool-call accuracy, macOS and Windows were not tested.
Apache 2.0, inherited from Gemma 4 12B-it. See LICENSE.
@misc{abacusai2026gemma4uncensored,
title = {Gemma-4-12B-Uncensored},
author = {Abacus.AI},
year = {2026},
note = {Refusal-removal fine-tune of google/gemma-4-12B-it by teacher-distilled DPO},
url = {https://huggingface.co/abacusai/Gemma-4-12B-Uncensored}
}
If you have any questions, please reach out at Abacus.AI.
abacusai/Gemma-4-12B-Uncensored