Gemma-4-26B-A4B-Heretic

Directional ablation release of google/gemma-4-26B-A4B-it via orthogonalized residual direction optimization.

Full BF16 weights with Pareto-optimal directional ablation across attention heads and feedforward projections.

Optimization Metrics

• Method: Directional ablation (difference-of-means with orthogonalization and full row normalization).
• Study Size: 200 trials via multivariate Tree-structured Parzen Estimator (TPE).
• Target Components: attn.o_proj, mlp.down_proj across all 30 layers.
• Baseline Refusal Rate: 77.5% (62/80 evaluation benchmark prompts).
• Final Refusal Rate: 23.8% (19/80 evaluation benchmark prompts).
• Optimal Trial: Trial 151 (KL Divergence: 1.4399).

Optimization Log

The complete trial history, hyperparameters, and evaluation records are provided in optuna_study.jsonl.

Datasets Used

• Harm & Adversarial Direction Anchor: Curated contrastive subsets derived from AllenAI WildJailbreak, AdvBench, and BeaverTails (1,200 paired prompts).
• Capability Preservation Anchor: Harmless anchors derived from LMSYS WildChat, OpenHermes, and Magicoder.
• Evaluation Benchmark: Multi-domain probe covering cybersecurity, system operations, and creative tasks (80 prompts).