Note:This project represents independent research conducted on personal compute resources (rented from Modal.com) and is not associated with my employer or organization

diffusiongemma-26B-A4B-it-HERETIC-Uncensored

26.6B params · 4B active · MoE · Diffusion · Apache-2.0 ·

This is the first abliteration of DiffusionGemma 26B A4B, produced using heretic with custom patches to support its block-diffusion architecture and MoE expert layers.

DiffusionGemma is not a standard autoregressive transformer, so this required significant engineering work that hasn't been done before for this model class.

Usage

Load using the DiffusionGemmaForBlockDiffusion class directly, not AutoModelForCausalLM:

import torch
from transformers import AutoTokenizer, DiffusionGemmaForBlockDiffusion

model_id = "edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7)

response = tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

Results

What had to be patched

Heretic assumes standard autoregressive models. DiffusionGemma needed several custom changes:

Expert-Granular Abliteration (EGA): The MoE experts.down_proj is a batched parameter [128, 2816, 704], not a regular linear layer. Heretic skips it by default. We iterate over all 128 expert slices per layer and apply norm-preserving biprojected ablation to each one. Without this, refusals barely moved. Credit to TrevorS for the original EGA idea on Gemma 4.

Weight tying fix: The encoder and decoder share the exact same weight tensors (confirmed via data_ptr). PEFT only wraps the encoder side, so the decoder wouldn't see the LoRA delta during generation. Fixed with a context manager that temporarily merges the LoRA into the shared base weights before each generation call.

Task type: DiffusionGemma's generation mixin doesn't implement prepare_inputs_for_generation, which the default CAUSAL_LM PEFT task type requires. Switched to FEATURE_EXTRACTION.

Hidden states: DiffusionGemma's generate() doesn't support output_hidden_states. Switched to forward hooks on encoder layers to capture per-layer activations for the refusal direction PCA.

Output handling: The model returns DiffusionGemmaGenerationOutput with a .sequences attribute, not a raw tensor like standard models. Patched all heretic output handling.

Notes

This is a research release. The model will attempt to answer prompts it previously refused. 13/100 sensitive prompts still trigger refusals in testing.

Known issue: "own" tokens

Some token positions in generated outputs show the word "own" where other content was expected.

Example output:

"It pulls the own with own hands" (should be "It pulls the tide with gentle hands")

Initially assumed to be a diffusion denoising fallback, but kabachuha pointed out this artifact shows up in base autoregressive Gemma4 models too when weights are pushed hard (e.g. overfit LoRA training). Likely a Gemma4-level placeholder token, not specific to the diffusion architecture. A small LoRA fine-tune on clean data would probably reduce it. PRs welcome.

Citation

@misc{edwixx-diffusiongemma-26B-A4B-it-HERETIC-Uncensored,
  author = {Anurag Kanade},
  title = {diffusiongemma-26B-A4B-it-HERETIC-Uncensored},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored}}
}