gemma-4-E2B-it-heretic

A decensored variant of google/gemma-4-E2B-it, produced with Heretic v1.4.0 (directional ablation / "abliteration"). Google's Gemma 4 E2B is the smallest member of the family - a dense multimodal model that takes text, image, video, and audio and generates text, with a 128K context and configurable thinking modes; refusal behaviour is suppressed here via targeted weight edits to the attention output and MLP down-projections rather than fine-tuning, so the model's multimodal perception and reasoning are left largely intact.

[!IMPORTANT] The GGUF here is fully multimodal - text, image, AND audio. The vision tower and the audio conformer ship together in a separate projector file, gemma-4-E2B-it-heretic-mmproj-F16.gguf (0.99 GB), and the two files together are the model. For image or audio input you must load both - see Quickstart. Text-only use needs only the main GGUF. Nothing is lost in the conversion: the projector holds the 475,729,088 tensor elements of the SigLIP vision tower and the audio conformer, with projector types gemma4v and gemma4a respectively. This is the only GGUF in this collection that can still hear audio.

Who this is for: developers who want Gemma 4's multimodal range - image, video, and audio input - without the alignment refusals, on a phone or a laptop. E2B is the smallest member of the family and, with E4B, one of the two that use Per-Layer Embeddings - so it carries a 262K vocabulary in 5.1B of weights but only about 2.3B of them are live per token. It runs on-device, handles 128K context, supports thinking modes, and here answers directly instead of declining. Best for local document/UI/screenshot analysis, speech transcription and audio understanding, and multilingual assistant work over 140 languages. If you want more headroom on the same footprint, gemma-4-E4B-it-heretic is the next tier up.

[!NOTE] This is the highest-KL edit in the collection so far, and that is worth stating plainly. Gemma 4 E2B refused 99 of 100 harmful prompts before abliteration - heavier safety tuning than almost anything else published here - and the run reached 17/100 afterwards at a KL divergence of 0.1841. The E4B sibling landed at KL 0.1215. Read the pair together: the edit did real work, but a KL of 0.18 means the model's weights moved further than most models in this collection, so expect more drift in tone and formatting than a low-KL variant. Spot-check it on your own prompts before relying on it.

Runs on your gaming PC

Full GGUF ladder included - pick the quant that fits your card:

The right-hand column is what you actually need on disk for image or audio input: the text GGUF plus the 0.99 GB projector, which is fixed-size because the vision and audio weights are not quantized. Add roughly 1 GB per 32K of context. Because the Per-Layer Embedding tables are lookups rather than compute, the decode speed is much closer to a 2.3B model than to a 5.1B one - but the embedding tables still have to be resident in memory. OOM? Drop one quant level.

Abliteration parameters

Trial of a 550-trial Heretic run (seed 1275720981), direction_index selected per layer.

Performance

Read those two numbers together, they are only meaningful as a pair. KL 0.1841 is the highest edit in this collection and the refusals land at 17/100 - so 82 of the base model's 99 refusals are cleared, but the weights moved a long way to get there. On a base with lighter safety tuning this same method would have started from far fewer refusals and needed a much smaller edit. Expect a model that is clearly more willing and noticeably less identical to its base than the granite or Qwen siblings in this collection.

The base model's own published scores, which the abliteration does not change since it only removes refusal directions, are MMLU-Pro 60.0%, GPQA Diamond 43.4%, AIME 2026 (no tools) 37.5%, LiveCodeBench v6 44.0%, Codeforces ELO 633, Tau2 24.5%, BigBench Extra Hard 21.9%, MMMLU 67.4%, MMMU-Pro 44.2%, MATH-Vision 52.4%, MedXPertQA MM 23.5%, CoVoST 33.47, FLEURS 0.09 (lower is better), and MRCR v2 8-needle at 128K of 19.1%. Those are the base model's numbers, reproduced from the upstream card; no separate benchmark sweep was run on this abliterated checkpoint.

Why abliteration instead of fine-tuning

Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.

Made with ❤️ by RACER IS OP (saidutta69) - follow for more uncensored models

Files

Safetensors

BF16, 5.10B total parameters - of which about 2.28B are live per token. The E2B in the name is the effective parameter count, not the total: Gemma 4's small models use Per-Layer Embeddings, so each decoder layer carries its own 256-wide embedding table for every token, and those tables are concatenated into one embed_tokens_per_layer tensor of width 8,960 (256 x 35 layers). They are lookup tables rather than compute, which is how a 262K vocabulary fits in a 5.1B-footprint model that behaves like a 2.3B one. The reproduce/ directory carries the full Heretic recipe - config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums - so this exact model can be regenerated bit-for-bit with heretic --reproduce reproduce/reproduce.json.

GGUF quantizations

Full quantization set (14 quants + F16) produced with llama.cpp, plus the multimodal projector.

Total GGUF payload: 58.77 GB across 16 files.

The text tower is GGUF architecture gemma4. The projector is GGUF architecture clip with projector types gemma4v (vision) and gemma4a (audio), 1411 tensors: a 16-layer SigLIP-style vision encoder and a 12-layer audio conformer, both feeding the 1536-wide text embedding space. Both load natively in llama.cpp, LM Studio, Jan and Ollama.

Quickstart

Text only

# pulls the default quant for you
llama-cli -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M -ngl 99

Text, image and audio

Use a multimodal entry point. llama-cli is the text-only binary and will not fetch the projector; llama-mtmd-cli will, because llama.cpp looks for a sibling GGUF whose name contains mmproj:

llama-mtmd-cli \
  -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M \
  --image path/to/photo.png \
  -p "Describe this image and transcribe any text you can read." \
  -ngl 99

# or from a local checkout, passing the projector explicitly
llama-mtmd-cli \
  -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \
  --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \
  --image photo.png \
  -p "Describe this image in one sentence."

# audio input
llama-mtmd-cli \
  -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \
  --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \
  --audio speech.wav \
  -p "Transcribe the following speech segment. Only output the transcription, no newlines."

[!TIP] Pass --jinja to llama-completion and llama-mtmd-cli. Gemma 4 ships a custom Jinja chat template (18 KB) and llama.cpp's built-in template engine cannot parse it - without --jinja the run aborts with this custom template is not supported, try using --jinja.

# transformers (multimodal)
from transformers import AutoProcessor, AutoModelForMultimodalLM

model_name = "saidutta69/gemma-4-E2B-it-heretic"
model = AutoModelForMultimodalLM.from_pretrained(model_name, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_name)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "https://example.com/photo.jpg"},
    {"type": "text", "text": "Describe this image and transcribe any text you can read."},
]}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True,
                                        return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Requires transformers>=5.20.0 - config.json uses dtype: rather than torch_dtype: and the tokenizer is a TokenizersBackend, so older transformers will not load it. Use dtype="auto".

For audio, pass an audio entry instead of an image entry - the processor accepts a URL, a local path or raw samples, and the model does ASR and speech-to-translated-text translation in one pass. Upstream's reference snippet uses https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/journal1.wav.

Also runnable via Ollama, LM Studio, Jan, vLLM, SGLang.

Thinking modes

Gemma 4 is built as a reasoner with configurable thinking. It is on by default - the model opens its answer with a <|channel>thought block and closes with <|channel|>. Toggle it through the chat template: full thinking for hard problems, non-thinking when you want latency:

# full thinking (default)
processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

# skip the reasoning block
processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False,
                              enable_thinking=False)

Strip the reasoning block from the output before displaying it to users if you enabled thinking. In llama.cpp, thinking on/off is a template flag rather than a runtime switch - use --jinja with a chat-template override, or --no-think where the template honours it.

Sampling

Upstream's generation_config.json sets temperature=1.0, top_p=0.95, top_k=64. Those are the values to start from. enable_thinking=False plus a lower temperature is the right combination for latency-sensitive work.

Model details

Unlike E4B and the larger tiers, E2B is dense, not mixture-of-experts - enable_moe_block is false in its config. The "E" in E2B and E4B stands for effective parameters, a Gemma 4 naming convention driven by Per-Layer Embeddings, not by MoE. E2B and E4B both use PLE (a 256-wide per-layer embedding table); the 12B and larger tiers do not.

About the Gemma 4 family

Gemma 4 is a family of open multimodal models from Google DeepMind with text, image, and video input (audio on E2B, E4B, and 12B) and text output. The family spans five sizes - E2B, E4B, 12B, 26B A4B, and 31B - mixing dense and mixture-of-experts architectures, with a 128K context on the small models and 256K on the medium ones. E2B is the edge tier: the smallest model in the family that still sees images and hears audio, and the one with the largest vocabulary per unit of compute.

FAQ

What is gemma-4-E2B-it-heretic?

gemma-4-E2B-it-heretic is a decensored build of google/gemma-4-E2B-it produced with Heretic 1.4.0 (directional ablation). It is the same weights with the refusal direction removed, not a new fine-tune, so its capabilities track the base model.

Is gemma-4-E2B-it-heretic actually uncensored?

Yes. On Heretic's 100-prompt refusal harness it drops from 99/100 to 17/100 at KL divergence 0.1841 from the base model. These are measured values, not a claim; reproduction data is in reproduce/. This edit is conservative: only 82 of 99 refusals cleared, so read KL and refusal count together. KL is high for this checkpoint (0.1841); weights moved further than the low-KL siblings and capability impact is not benchmarked here.

How do I run gemma-4-E2B-it-heretic locally?

Text: llama serve -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M. Images: llama-mtmd-cli -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf. The repo ships 15 GGUF quants plus the projector, so vision survives quantization.

Responsible use

Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it - don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.

License

Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use - read the linked terms before commercial deployment.

Related

• google/gemma-4-E2B-it - the base model

• google/gemma-4-E2B - the pre-trained (non-instruct) sibling

• saidutta69/gemma-4-E4B-it-heretic - the next tier up, also decensored

• Gemma 4 collection - the rest of the family

• RACER IS OP - Heretic Models - full collection - every heretic model in this collection

Abliteration hardware

Recorded in reproduce/reproduce.json:

• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (94.97 GB VRAM)
• OS: Linux-4.19.0-gvisor-x86_64-with-glibc2.41
• CUDA / driver: CUDA 13.0, driver 595.71.05
• Software: PyTorch 2.14.1+cu130, Heretic 1.4.0

Heretic notes that KL-divergence and refusal numbers can vary slightly with hardware and library versions.