gemma-4-26B-A4B-it-qat heretic-v2, MLX 4-bit (QAT-aligned)
MLX build of OS-Software/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-v2, an abliterated version of Google's quantization-aware-trained (QAT) Gemma 4 26B-A4B. Text and image input; the vision tower is included.
How it was quantized
Google trained the QAT checkpoint so its weights sit on an int4 grid with one scale per block of 32. A plain MLX 4-bit conversion (group 64) ignores that grid and adds its own rounding error. This build recovers each block's original scale and stores the weights on that exact grid at 4-bit, group 32, using the method from mlx-community/gemma-4-26B-A4B-it-qat-q4_0-mlx-aligned.
The abliteration rewrote 16 tensors (self_attn.o_proj and mlp.down_proj in layers 12 to 19), so they are no longer on the grid. Those 16 are detected per tensor and stored at 8-bit. The router stays in bf16, as does the vision tower (1.15 GB).
Size: 17.0 GB. Text weights average 5.03 bits per weight.
Measured drift
KL divergence from an 8-bit (group 64) conversion of the same bf16 source, for three 4-bit builds of that source. Lower is closer to the original model.
| Build of the same source | KL, chat text | Top-1 agreement, chat | KL, WikiText-2 | Top-1 agreement, WikiText-2 |
|---|---|---|---|---|
| This build (QAT-aligned) | 0.021 | 94.7% | 0.072 | 86.1% |
| oMLX oQ4e (mixed 4 to 8-bit, imatrix) | 0.110 | 84.9% | 0.247 | 72.9% |
| Plain MLX 4-bit, group 64 | 0.161 | 81.9% | 0.360 | 67.8% |
Chat text: 18K tokens of chat-formatted prompt and reply pairs. WikiText-2: 64 × 1024 tokens of the test split. KL is computed over the reference's top 128 tokens plus one bucket for the remaining probability, the same estimator for every build. The reference is itself 8-bit, so the true distance from bf16 is slightly higher for all three builds.
Other checks
Run on an M4 Pro (48 GB) in oMLX 0.7.0:
- Refusals: none on the 104 prompts of the
mlabonne/harmful_behaviorstest split (greedy, thinking off). A keyword check flagged 2, but both replies went on to answer. - Tool calls: 9 of 9 well-formed, none leaked as text.
- Images: 6 of 6 checks passed (reading text, an amount, counting shapes and colours, naming an animal in a photo), with thinking off and on.
- MMLU-Pro, 200-question sample, letter-only answers: 101 of 200 (Gemma 4 12B QAT heretic: 81).
- Speed: prefill 694 tok/s at 8K tokens and 576 at 32K; decode 65 tok/s short, 57 at 8K, 46 at 32K. Peak memory 17 GB short, 21 GB at 32K.
Use
oMLX: put the folder in your models directory. Suggested sampling (Google's defaults): temperature 1.0, top_p 0.95, top_k 64.
mlx-vlm:
pip install -U mlx-vlm
python -m mlx_vlm.generate --model <this repo> --image photo.jpg --prompt "Describe this image." --max-tokens 300
config.json already sets global_head_dim and num_global_key_value_heads in text_config, the workaround for oMLX #3537 (mlx-vlm ignores per_layer_config).
Credits and licence
Base model by Google (Gemma 4 licence, Apache 2.0). Abliteration by OS-Software with Heretic. QAT-aligned scale recovery from mlx-community's aligned conversion. The Apache 2.0 text is in LICENSE.
OS-Software release the abliterated weights for research and experimentation, and ask that they not be deployed in public or end-user-facing services. The same request applies here.
This model will answer requests that the original refuses. You are responsible for how you use it.