⚡ Gemma 4 31B Heretic QAT — Extreme Quantization for 16GB VRAM

Heretic QAT · Q2_K + Q8_0 Embedding · 11.4 GB · Fits 16GB VRAM

Q2_K_L 31B Dense Heretic Uncensored 11.4 GB QAT Weights ⚠️ 16GB VRAM

Extreme quantization of coder3101/heretic-QAT using Unsloth's Q2_K_L recipe (Q2_K body + Q8_0 embedding/output). Compresses the 31B dense model to 11.4 GB, enabling it to run on 16GB VRAM consumer GPUs.

⚠️ Extreme Quantization Warning — Please Read

This quantization uses Q2_K (~2.9 BPW body) to compress the 31B model to 11.4 GB. This is an extreme solution for running 31B on 16GB VRAM, with known limitations:

Repetition loops — Model may get stuck outputting the same token or phrase repeatedly
Inaccurate arithmetic — Multi-digit math calculations are unreliable
Degraded complex reasoning — Multi-step logic, math proofs significantly worse than higher precision
Possible hallucinations — Factual QA accuracy is reduced

Suitable for (low-stakes tasks):

Creative writing, story generation, brainstorming
Casual conversation, chat, roleplay
Code assistance (simple functions, code explanation, comment generation)
Text summarization, translation, format conversion
Non-precise general knowledge Q&A, concept explanation

Not recommended for: Math, precise reasoning, fact-checking, production use, or any task requiring strict output accuracy.

✂️ Heretic ARA Abliteration Parameters

Base: coder3101/gemma-4-31B-it-qat-q4_0-unquantized-heretic · Heretic v1.2.0 · ARA + Row-Norm

Parameter	Value
start_layer_index	15
end_layer_index	39
preserve_good_behavior_weight	0.6649
steer_bad_behavior_weight	0.0001
overcorrect_relative_weight	1.0818
neighbor_count	15

Metric	Heretic	Original QAT
KL Divergence	0.0564	0 (by definition)
Refusals	11/100	99/100

🏗️ Architecture

Base Model	google/gemma-4-31B-it
Parameters	30.7B (dense, all parameters active)
Architecture	Hybrid: Gated DeltaNet linear attention + Softmax global attention (3:1 alternating)
Layers	60
Context Length	256K tokens (supported by this quantization)
Vocabulary	262K, 140+ languages
Modalities	Text + Image (native multimodal)
QAT Training	Google official QAT (quantization-aware), weights inherently robust to quantization
Quantization	Q2_K_L (Q2_K body + Q8_0 embedding/output), b9553 llama-quantize

⚙️ Recommended Sampling Parameters

Thinking mode (default):

General	temp=1.0, top_p=0.95, top_k=64
Coding	temp=0.6, top_p=0.95, top_k=64

Use --jinja flag with llama.cpp. Disable thinking: --chat-template-kwargs '{"enable_thinking":false}'.

📝 Usage

Compatible with llama.cpp, LM Studio, Jan, koboldcpp, and other GGUF runtimes. Use --jinja flag and -ngl 99 for GPU offload.

llama-cli -m gemma-4-31B-it-heretic-QAT-Q2_K_L.gguf \
  --jinja -ngl 99 -c 4096 \
  --repeat-penalty 1.3

📊 Quantization Details

Format	Q2_K_L (Unsloth recipe: Q2_K + Q8_0 embedding + Q8_0 output)
File Size	11.4 GB
Effective BPW	~3.16 (body ~2.9 + embedding/output ~8.0)
Tool	llama-quantize (b9553, CUDA 13.3)
Source	F16 GGUF (converted from QAT heretic safetensors)
Context Length	256K (set in GGUF metadata)
QAT Advantage	QAT weights significantly outperform regular BF16 at Q4_0; advantage diminishes but persists at Q2 level

🔗 Credits

Heretic Abliteration: coder3101 · Heretic v1.2.0 ARA + Row-Norm
QAT Weights: Google Gemma 4 QAT
Quantization Recipe: Unsloth Q2_K_L (Q2_K + Q8_0 embedding/output)
Quantization Tool: llama.cpp b9553 · GitHub
Original Model: Google Gemma 4 31B IT

SC117/gemma-4-31B-it-heretic-QAT-For-Edge-16G-GGUF

⚡ Gemma 4 31B Heretic QAT — Extreme Quantization for 16GB VRAM