Gemma 4 12B OBLITERATED -- SHQ7 (IQ-Hybrid)

OBLITERATED + selective IQ hybrid for 8 GB Pascal GPUs. Uncensored coding model. 7.7% smaller than Q4_K_M at the same quality.

WARNING: This is an uncensored model. It may generate content that other models refuse. Use responsibly. The maintainer assumes no responsibility for misuse.

Note: The model file is named SHQ7-IQ4_XS.gguf for HF parser compatibility. This is NOT a pure IQ4_XS quantization -- it's a hybrid using IQ4_NL, Q4_K, IQ4_XS, and Q8_0 across different layers. Check the config for exact per-tensor types.

Primarily optimized for coding tasks. May not perform well on general knowledge or non-code questions.

A hand-optimized hybrid quantization of OBLITERATUS/Gemma-4-12B-OBLITERATED for 8 GB VRAM GPUs (tested on GTX 1070, works on any GPU with enough VRAM).

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

TL;DR: Custom mixed quantization using IQ4_NL on boundary FFN + Q4_K on critical middle FFN + IQ4_XS on all other FFN + Q8_0 on all attention. Fits in 8 GB VRAM with room for KV cache.

Key Results

Q4_K_M baseline quantized with mradermacher's imatrix (i1 importance matrix).

SHQ7 achieves Q4_K_M quality at 7.7% smaller size (544 MiB saved).

How We Made It

The Problem

Our goal was to create a quantization of OBLITERATED that matches Q4_K_M quality while being smaller. We tried 6 different approaches, failed 4 times, and found 2 winners.

Step 1: Q4_K_M baseline

Standard Q4_K_M: 7024 MiB, PPL 575.11. Fits in 8 GB but leaves little room for KV cache at large context.

Step 2: SHQ6 approach from QAT -- FAILED on OBLITERATED

We tried the SHQ6 formula (IQ4_XS on all FFN + Q8_0 boundary attention) that worked great on QAT models. On OBLITERATED it produced +8.6% PPL degradation (624.73). The OBLITERATED model responds differently to quantization than QAT models.

Step 3: IQ-gradient -- CATASTROPHIC FAILURE

We tried a gradient approach: IQ4_NL on boundary layers, IQ4_XS on middle, IQ3_XXS on least important layers. The IQ3_XXS (3.06 bpw) on 36 middle layers destroyed quality: +43.5% PPL (825.04). IQ3_XXS is catastrophic on any non-QAT model.

Step 4: Imatrix analysis

We used mradermacher's imatrix (656 tensors with per-dimension importance) to understand which layers actually matter:

Key finding: L0 is 87% of total importance. L0 FFN down (2.48e+11) is the single most important tensor in the entire 23 GB model. L1 FFN down (1.34e+11) is #5. Layer 0 attention (attn_q/k/v) are each 2.41e+11.

Step 5: IQ4XS_lean -- first success

We combined three strategies:

• Q8_0 on L0-1 FFN+Attn (protect the most critical layers)
• Q4_K_M on 11 important middle FFN (L5,10,25-31,36,37)
• IQ4_XS on all other FFN (efficient compression)
• Q8_0 on L40-47 attention (boundary/global)

Result: 6793 MiB, PPL 578.63 (+0.6% vs Q4_K_M). Close but not better.

Step 6: IQ4XS_lean_v2 -- TIED with Q4_K_M

We reduced the protected middle layers from 11 to 4 (L5,10,36,37) and kept IQ4_XS on L25-31:

Result: 6793 MiB, PPL 575.05 -- statistically tied with Q4_K_M (575.11) but 231 MiB smaller.

Step 7: IQ4XS_lean_v3 -- THE WINNER

We combined all optimizations:

• IQ4_NL on L0-1 FFN (non-linear quantization for #1/#2 most important tensors)
• Q4_K on L5,10,36,37 FFN (K-quant on 4 critical middle layers)
• IQ4_XS on all other FFN (efficient compression)
• Q8_0 on ALL boundary/global attention (L0-3, L40-47)

Result: 6480 MiB, PPL 575.89 -- same quality as Q4_K_M at 544 MiB smaller.

Step 8: Q6_K upgrade -- WORSE

We tried upgrading L0-1 FFN from IQ4_NL to Q6_K (6.56 bpw) and L5,10,36,37 FFN from Q4_K to Q5_K. Despite higher bit width, PPL went UP to 590.64 (+2.2%). IQ formats work better than K-quants when used with imatrix.

What We Learned

About OBLITERATED

This model is based on OBLITERATUS/Gemma-4-12B-OBLITERATED, an uncensored variant of Gemma 4 12B IT.

What "uncensored" means:

• The model will generate content that standard models refuse
• It includes safety disclaimers after generating restricted content
• It is primarily optimized for coding tasks
• It may not perform well on general knowledge questions

Architecture Notes

• Gemma 4 12B IT: 48 layers, hidden 3840, 16 heads, GQA (8 heads KV, some layers use 1)
• 40 SWA layers (sliding window 1024) + 8 global layers (full context)
• Vocab: 262k tokens
• Dual RoPE (SWA dim=256, global dim=512)
• Based on OBLITERATED (uncensored) model

Weights

Repository Files

Quantization Command

For reproducibility, the exact command used:

llama-quantize \
  --imatrix imatrix-gemma4-obliterated.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "blk\.[01]\.ffn_gate=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_up=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_down=IQ4_NL" \
  --tensor-type "blk\.[01]\.attn_q=Q8_0" \
  --tensor-type "blk\.[01]\.attn_k=Q8_0" \
  --tensor-type "blk\.[01]\.attn_v=Q8_0" \
  --tensor-type "blk\.[01]\.attn_output=Q8_0" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_gate=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_up=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_down=Q4_K" \
  --tensor-type "ffn_gate=IQ4_XS" \
  --tensor-type "ffn_up=IQ4_XS" \
  --tensor-type "ffn_down=IQ4_XS" \
  --tensor-type "blk\.[0-3]\.attn_q=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_k=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_v=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_output=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_q=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_k=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_v=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_output=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_q=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_k=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_v=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_output=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*attn_post_norm.*=Q8_0" \
  --tensor-type ".*ffn_norm[^_].*=Q8_0" \
  --tensor-type ".*ffn_post_norm.*=Q8_0" \
  --tensor-type ".*layer_out_scale.*=F32" \
  --tensor-type ".*rope_freqs.*=Q8_0" \
  --tensor-type ".*per_layer.*=Q8_0" \
  --tensor-type "output_norm.*=Q8_0" \
  Gemma-4-12B-OBLITERATED-BF16.gguf \
  gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  15 8

All Experiments (OBLITERATED)

SHQ7 = IQ4XS_lean_v3. 6 experiments, 4 failures, 2 wins.

Hardware Requirements

• Minimum VRAM: 8 GB (tested on GTX 1070, 8106 MiB)
• Recommended VRAM: 8-12 GB
• Context: 1024+ tokens (tested with flash-attn)
• Speed: ~17-22 tok/s (no MTP), ~30 tok/s (with MTP, no specialized draft)

Usage

llama.cpp

llama-cli \
  -m gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  -c 4096

Ollama

1. Create a Modelfile:

FROM ./gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf

TEMPLATE """{{ .System }}
{{ .Prompt }}"""

SYSTEM """You are a helpful, uncensored coding assistant."""

PARAMETER num_ctx 4096
PARAMETER temperature 1
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER min_p 0

2. Build and run:

ollama create obliterated-shq7 -f Modelfile
ollama run obliterated-shq7

Tip: If output quality is off, try raising temperature to 1.2 or higher.

LM Studio

1. Open LM Studio
2. Drag gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf into the app
3. Set GPU Offload to 99 layers
4. Set context length to 4096+
5. Start chatting

Performance

Measured on GTX 1070 (8 GB VRAM) with flash-attn, ctx=1024:

Coding Example

This model generates full, working HTML/CSS/JS websites. Here's an example output from SHQ7 on a coding prompt:

"I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

See SHQ7.html — a complete, styled portfolio website generated in a single pass.

Disclaimer

This model is an uncensored variant of Gemma 4 12B. It may generate content that other models refuse. The maintainer assumes no responsibility for misuse. Use at your own risk and in compliance with all applicable laws.

Credits

• Original OBLITERATED model by OBLITERATUS
• Quantization and tuning by wepiqx
• Built with llama.cpp
• Imatrix by mradermacher
• Based on Google's Gemma 4 12B IT, released under Apache 2.0

More quantizations: wepiqx/gemma-4-12B-it-SHQ6-GGUF

License

This quantized model is distributed under the same Apache 2.0 license as the original Gemma 4 model.