gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara — oQ4e
Mixed-precision quant of mewse/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara, produced with oQ (oMLX v0.6.4). Vision tower intact — image input still works. Standard MLX safetensors — compatible with oMLX, mlx-vlm, and any MLX-capable app.
What is oQ?
Unlike uniform 4-bit quantization, oQ is a data-driven mixed-precision quantizer that calibrates per-layer sensitivity and allocates bits where they matter most. Critical layers (embeddings, the most sensitive transformer layers) are automatically promoted to higher precision, while less sensitive layers stay at 4-bit.
In this quant: embed_tokens at 8-bit, sensitive down_proj / attention projections at 5–6-bit, final layers at 8-bit, everything else 4-bit (group size 64).
Known loading issue (already handled in this repo)
oMLX #3537: the pinned mlx-vlm Gemma4 loader ignores text_config.per_layer_config (per-layer KV head overrides on the full-attention layers), so loading fails with Expected (4096, 440) received (1024, 440).
The config.json in this repo already ships the workaround — global_head_dim: 512 and num_global_key_value_heads: 2 are present in text_config, with per_layer_config left in place. The model loads as-is on oMLX 0.6.4; no manual patching needed. (If you re-quantize or re-export from another copy of this checkpoint, re-apply those two fields.)
Quick start
# oMLX
omlx serve --model underlotus/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara-oQ4e
# mlx-vlm (multimodal — Gemma4ForConditionalGeneration)
from mlx_vlm import load
model, tokenizer = load("underlotus/gemma-4-26B-A4B-it-qat-q4_0-unquantized-heretic-ara-oQ4e")
Original model
- Base: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized (QAT bf16 weights)
- Decensored: Heretic v1.2.0 with Arbitrary-Rank Ablation (ARA, PR #211) with row-norm preservation — KL divergence 0.1226, 5/100 refusals (bf16 sweep; 10/100 at IQ4_XS)
- Abliteration params: layers 11–23, preserve 0.9713 / steer 0.0037 / overcorrect 0.9798
- Architecture: MoE — 26B total / ~4B active, 128 experts (top-8), 30 layers, 262144 context
- Multimodal: image + text input (26B A4B has no audio); ships the canonical 2026-07 chat template in
chat_template.jinja
License
Gemma license (Apache 2.0 with Gemma use restrictions), inherited from the base model.