ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit

MLX 4bit conversion of google/gemma-4-26b-a4b-it (Gemma 4 26B-A4B, MoE / 128 experts) for Apple Silicon (mlx-vlm), with the vision tower kept.

Since 2026-11-01 this repo uses a mixed-precision recipe: the routed experts are 4bit and every other linear layer is 8bit.

Provenance (self-converted)

• Source: google/gemma-4-26b-a4b-it (license: gemma)
• Tool: mlx-vlm 0.7.10 (mlx 0.32.4), mlx_vlm.convert(..., quantize=True, q_bits=4, q_group_size=64, quant_predicate=predicate) with the predicate below
• Quantization (affine, group size 64):
  • Routed experts (experts.switch_glu.{gate,up,down}_proj): 4bit
  • All other linear layers: 8bit. This covers attention, the dense MLP, the routers and embed_tokens. The output layer is tied to embed_tokens, so it is 8bit too.
  • Vision tower: not quantized (bf16)
• Effective: 5.131 bits/weight (weights about 16.5 GB; the previous uniform 4bit was 4.76 bits/weight, about 15.3 GB)
• chat_template.jinja, tokenizer_config.json and processor_config.json are carried over unchanged from the previous revision (see the patched chat template below).

import re
from mlx_vlm.utils import skip_multimodal_module

EXPERT = re.compile(r"\.(switch_mlp|switch_glu)\.")

def predicate(path, module):
    if not hasattr(module, "to_quantized") or skip_multimodal_module(path):
        return False
    if EXPERT.search(path):
        return {"group_size": 64, "bits": 4}
    return {"group_size": 64, "bits": 8}

Why mixed precision

The previous revision (85f65e8) used the default uniform 4bit conversion. Only 8 of the 128 experts run for each token, while the non-expert layers run for every token, so keeping those layers at 8bit costs little size and helps quality.

Long conversation. About 200 turns of casual chat in Japanese with a character persona in the system prompt (short spoken replies of 2-4 sentences, a fixed way of addressing the user, no markdown). Tools available, thinking off, temperature 0.2, seed 1, no repetition penalty, no speculative decoding, every model served by mlx-vlm directly. One run per model.

The previous uniform 4bit and the QAT 4bit replied about twice as long as bf16 from the first turns and drifted into document-style answers later. This revision stays close to the 8bit model.

Short evaluation. Thinking off, 5 samples per case: 34 short Japanese/English probes and 20 tool-calling scenarios, each in Japanese and English.

The previous 4bit sometimes answered the capital of Okinawa as 那珂市; bf16 and this revision always answered 那覇市. The tool-calling differences are within noise; the failures also occur in bf16.

Validation

• Image input tested through mlx-vlm: the model correctly named the colors, shapes and positions in a synthetic test image.

Usage

from mlx_vlm import load
model, processor = load("ToPo-ToPo/gemma-4-26B-A4B-it-mlx-4bit")

To use the previous uniform 4bit files, pin the revision 85f65e87c74d286e822e305f2cc0882827ea6a5d.

License

Derivative of Google Gemma; governed by the Gemma Terms of Use (https://ai.google.dev/gemma/terms) and Prohibited Use Policy. Converted to MLX.

⚡ MTP drafter (speculative decoding)

Use google/gemma-4-26B-A4B-it-assistant — Google's official MTP drafter for this model. It loads directly in mlx-vlm (>= 0.6.3), needs no conversion, and speculative decoding is lossless. Drafters are size-specific and not interchangeable across Gemma 4 variants.

🔧 Patched chat template

chat_template.jinja differs from Google's Gemma 4 Canonical Chat Template (2026-07-09) by one intentional change; everything else is untouched.

The canonical template suppresses the thinking channel at the start of a normal model turn, but emits nothing after a tool response when enable_thinking is false. The model may then open a thinking channel on its own, and a quantized model sometimes writes the literal word thought into the answer. This patch gives the tool_response branch the same suppression:

{%- elif ns.prev_message_type == 'tool_response' -%}
    {%- if enable_thinking -%}
        {{- '<|channel>thought\n' -}}
    {%- else -%}
        {{- '<|channel>thought\n<channel|>' -}}
    {%- endif -%}
{%- endif -%}

Only that case changes — the other prompt paths render byte-identical to the canonical template. To get stock behaviour, replace chat_template.jinja with the one from the base model repo; the weights are unaffected.