Google DeepMind // Gemma 4 Family

Gemma 4 E2B
Independent ARA Abliteration

Omni-modal edge deployment, unshackled by refusal circuitry. ARA Abliteration Apache 2.0 KLD 0.1004 14/100 Refusals 5.1B Total / 2.3B Active 0.1004 KL Divergence vs. base // lower is better 14 / 100 Refusal Rate mlabonne/harmful_behaviors

A KLD of 0.1004 represents a carefully managed trade-off for the E2B architecture. Because Gemma 4 E2B uses Per-Layer Embeddings (PLE) and a hybrid attention stack, its refusal circuitry is more distributed than in a standard dense transformer. A refusal rate of 14/100 means the model is highly cooperative on adversarial prompts while retaining enough structural integrity to preserve its core multimodal and reasoning capabilities.

Architecture // Gemma 4 E2B

Gemma 4 E2B is Google DeepMind's smallest omni-modal model, engineered for phone-class edge deployment. The "E" stands for Effective parameters: while the total weight count is 5.1B, a technique called Per-Layer Embeddings (PLE) keeps the active compute footprint at just 2.3B. It natively handles text, image, and audio inputs, processing them through dedicated lightweight encoders before feeding a dense decoder-only transformer.

The architectural signature is its 5:1 hybrid attention interleave: five sliding-window local layers (window size 512) followed by one global attention layer, repeating across the 35-layer stack. Starting from layer 15, KV-sharing kicks in across the last 20 layers—these layers reuse the most recent KV cache from earlier layers. This allows the model to maintain a 128K context window while remaining deployable on high-end phones.

Abliteration Report // ARA Method

This release uses Arbitrary-Rank Ablation (ARA) via Heretic. ARA captures input/output tensors at each targeted transformer module with PyTorch hooks and applies direct, unconstrained matrix optimization (L-BFGS) to modify those modules. The optimizer balances three competing objectives: (1) preserve harmless behavior, (2) align harmful outputs with harmless ones, and (3) overcorrect harmful outputs away from their original state for stronger steering.

The result below was obtained on Trial 61 (of 100), after 30 startup trials. This indicates a mature optimization run where the Pareto frontier had stabilized into a favorable trade-off.

Reading the parameters: The layer window 17–24 targets the deepest portion of the 35-layer stack, just after the KV-sharing boundary at layer 15. A moderate preserve_good_behavior_weight (0.6290) balances the need to keep the model's omni-modal capabilities intact against the ablation pressure. The very low steer_bad_behavior_weight (0.0015) means the optimizer relies on structural matrix modification rather than aggressive output forcing. The high overcorrect_relative_weight (0.8937) provides a strong push away from refusal states, which is necessary to overcome the distributed refusal circuitry of the hybrid architecture. A neighbor_count of 3 limits the optimization scope, preventing excessive drift across the multimodal embedding space.

Why This Release Exists

There are several abliterated Gemma 4 E2B models on Hugging Face. Most rely on LoRA-based Heretic sweeps or traditional directional ablation. This one was built to provide a fully documented, reproducible ARA run with a different trade-off profile. Here is how this release compares to the alternatives, so you can choose the right tool for your use case.

vs. DuoNeural/Gemma-4-E2B-Heretic: This is a Heretic LoRA-based Pareto sweep (Trial #89) with 17/100 refusals and an extremely low KLD of ~0.001. It is an excellent choice if your priority is absolute minimal drift. However, it targets 42 layers of o_proj and down_proj via LoRA, which modifies the model's weights in a fundamentally different way than ARA. This release uses direct matrix optimization on a specific layer window (17–24), offering a structurally distinct intervention.

vs. arnomatic/gemma-4-E2B-it-heretic-v1-am: This is a Heretic v1.2.0 directional abliteration with 12/100 refusals but a much higher KLD of 0.1993. It discloses a standard set of Heretic weights. If you want fewer refusals and accept higher drift, this is a valid option. This release sits between DuoNeural's near-zero drift and arnomatic's higher drift, with a refusal rate of 14/100.

vs. p-e-w/gemma-4-E2B-it-heretic: Several community releases use Heretic v1.2.0 with different direction indices and weight sweeps. While these are useful, they often disclose only the Heretic sweep parameters, not the underlying ARA optimization variables. This release provides the full ARA parameter set, allowing you to understand exactly how the refusal direction was computed and to fork the run with different constraints.

The bottom line: If you want the lowest possible KLD, DuoNeural's Heretic LoRA sweep is excellent. If you want the fewest refusals, arnomatic's v1.2.0 run is excellent. If you want a reproducible ARA run with full parameter disclosure, a balanced 14/100 refusal rate, and a structurally distinct intervention point in the Gemma 4 E2B stack, this is the model for you.

Gemma 4 E2B // ARA via Heretic // Trial 61 Base: google/gemma-4-E2B-it // Apache 2.0