Sign up

underlotus/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara-oQ4e

🤗 Hugging Face sourceimage-text-to-textapache-2.025.8B params52 GBsafetensors✓ HF checksums availableupdated today
No torrent yet →

gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara — oQ4e

Mixed-precision quant of mewse/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara, produced with oQ (oMLX v0.6.4). Vision tower intact — image input still works. Standard MLX safetensors — compatible with oMLX, mlx-vlm, and any MLX-capable app.

What is oQ?

Unlike uniform 4-bit quantization, oQ is a data-driven mixed-precision quantizer that calibrates per-layer sensitivity and allocates bits where they matter most. Critical layers (embeddings, the most sensitive transformer layers) are automatically promoted to higher precision, while less sensitive layers stay at 4-bit.

In this quant: embed_tokens at 8-bit, sensitive down_proj / attention projections at 5–6-bit, final layers at 8-bit, everything else 4-bit (group size 64).

Known loading issue (already handled in this repo)

oMLX #3537: the pinned mlx-vlm Gemma4 loader ignores text_config.per_layer_config (per-layer KV head overrides on the full-attention layers), so loading fails with Expected (4096, 440) received (1024, 440).

The config.json in this repo already ships the workaround — global_head_dim: 512 and num_global_key_value_heads: 2 are present in text_config, with per_layer_config left in place. The model loads as-is on oMLX 0.6.4; no manual patching needed. (If you re-quantize or re-export from another copy of this checkpoint, re-apply those two fields.)

Quick start

# oMLX
omlx serve --model underlotus/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara-oQ4e

# mlx-vlm (multimodal — Gemma4ForConditionalGeneration)
from mlx_vlm import load

model, tokenizer = load("underlotus/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara-oQ4e")

Original model

  • Base: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (QAT bf16 weights)
  • Decensored: Heretic v1.2.0 with Arbitrary-Rank Ablation (ARA, PR #211) with row-norm preservation — KL divergence 0.1226, 5/100 refusals (bf16 sweep; 10/100 at IQ4_XS)
  • Abliteration params: layers 11–23, preserve 0.9713 / steer 0.0037 / overcorrect 0.9798
  • Architecture: MoE — 26B total / ~4B active, 128 experts (top-8), 30 layers, 262144 context
  • Multimodal: image + text input (26B A4B has no audio); ships the canonical 2026-07 chat template in chat_template.jinja

License

Gemma license (Apache 2.0 with Gemma use restrictions), inherited from the base model.