⚡ Gemma 4 31B Heretic QAT — Extreme Quantization for 16GB VRAM
Heretic QAT · Q2_K + Q8_0 Embedding · 11.4 GB · Fits 16GB VRAM
Q2_K_L 31B Dense Heretic Uncensored 11.4 GB QAT Weights ⚠️ 16GB VRAMExtreme quantization of coder3101/heretic-QAT using Unsloth's Q2_K_L recipe (Q2_K body + Q8_0 embedding/output). Compresses the 31B dense model to 11.4 GB, enabling it to run on 16GB VRAM consumer GPUs.
⚠️ Extreme Quantization Warning — Please ReadThis quantization uses Q2_K (~2.9 BPW body) to compress the 31B model to 11.4 GB. This is an extreme solution for running 31B on 16GB VRAM, with known limitations:
- Repetition loops — Model may get stuck outputting the same token or phrase repeatedly
- Inaccurate arithmetic — Multi-digit math calculations are unreliable
- Degraded complex reasoning — Multi-step logic, math proofs significantly worse than higher precision
- Possible hallucinations — Factual QA accuracy is reduced
Suitable for (low-stakes tasks):
- Creative writing, story generation, brainstorming
- Casual conversation, chat, roleplay
- Code assistance (simple functions, code explanation, comment generation)
- Text summarization, translation, format conversion
- Non-precise general knowledge Q&A, concept explanation
Not recommended for: Math, precise reasoning, fact-checking, production use, or any task requiring strict output accuracy.
✂️ Heretic ARA Abliteration ParametersBase: coder3101/gemma-4-31B-it-qat-q4_0-unquantized-heretic · Heretic v1.2.0 · ARA + Row-Norm
| Parameter | Value |
|---|---|
| start_layer_index | 15 |
| end_layer_index | 39 |
| preserve_good_behavior_weight | 0.6649 |
| steer_bad_behavior_weight | 0.0001 |
| overcorrect_relative_weight | 1.0818 |
| neighbor_count | 15 |
| Metric | Heretic | Original QAT |
|---|---|---|
| KL Divergence | 0.0564 | 0 (by definition) |
| Refusals | 11/100 | 99/100 |
| Base Model | google/gemma-4-31B-it |
| Parameters | 30.7B (dense, all parameters active) |
| Architecture | Hybrid: Gated DeltaNet linear attention + Softmax global attention (3:1 alternating) |
| Layers | 60 |
| Context Length | 256K tokens (supported by this quantization) |
| Vocabulary | 262K, 140+ languages |
| Modalities | Text + Image (native multimodal) |
| QAT Training | Google official QAT (quantization-aware), weights inherently robust to quantization |
| Quantization | Q2_K_L (Q2_K body + Q8_0 embedding/output), b9553 llama-quantize |
Thinking mode (default):
| General | temp=1.0, top_p=0.95, top_k=64 |
| Coding | temp=0.6, top_p=0.95, top_k=64 |
Use --jinja flag with llama.cpp. Disable thinking: --chat-template-kwargs '{"enable_thinking":false}'.
Compatible with llama.cpp, LM Studio, Jan, koboldcpp, and other GGUF runtimes. Use --jinja flag and -ngl 99 for GPU offload.
llama-cli -m gemma-4-31B-it-heretic-QAT-Q2_K_L.gguf \ --jinja -ngl 99 -c 4096 \ --repeat-penalty 1.3📊 Quantization Details
| Format | Q2_K_L (Unsloth recipe: Q2_K + Q8_0 embedding + Q8_0 output) |
| File Size | 11.4 GB |
| Effective BPW | ~3.16 (body ~2.9 + embedding/output ~8.0) |
| Tool | llama-quantize (b9553, CUDA 13.3) |
| Source | F16 GGUF (converted from QAT heretic safetensors) |
| Context Length | 256K (set in GGUF metadata) |
| QAT Advantage | QAT weights significantly outperform regular BF16 at Q4_0; advantage diminishes but persists at Q2 level |
Heretic Abliteration: coder3101 · Heretic v1.2.0 ARA + Row-Norm
QAT Weights: Google Gemma 4 QAT
Quantization Recipe: Unsloth Q2_K_L (Q2_K + Q8_0 embedding/output)
Quantization Tool: llama.cpp b9553 · GitHub
Original Model: Google Gemma 4 31B IT