ToPo-ToPo/gemma-4-31b-it-qat-mlx-4bit

MLX 4bit conversion of google/gemma-4-31B-it-qat-q4_0-unquantized for Apple Silicon (mlx-vlm).

Provenance (self-converted from official weights)

• Source: google/gemma-4-31B-it-qat-q4_0-unquantized (license: gemma)
• Tool: mlx-vlm 0.6.3 — mlx_vlm.convert --hf-path google/gemma-4-31B-it-qat-q4_0-unquantized --mlx-path . -q --q-bits 4 --q-group-size 64
• Effective: 4.713 bits/weight
• Validation: reproduced geometrically exact CAD output in an agentic CAD+FEM pipeline (volumes match the reference mlx-community conversion).

Usage

from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/gemma-4-31b-it-qat-mlx-4bit")

License

This is a derivative of Google Gemma. Use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy. Weights were converted/quantized to MLX format (modification notice per the Gemma Terms).

⚡ MTP drafter (speculative decoding)

Use google/gemma-4-31b-it-assistant — Google's official MTP drafter for this model. It loads directly in mlx-vlm (>= 0.6.3), needs no conversion, and speculative decoding is lossless. Drafters are size-specific and not interchangeable across Gemma 4 variants.

🔧 Patched chat template

chat_template.jinja differs from Google's Gemma 4 Canonical Chat Template (2026-07-09) by one intentional change; everything else is untouched.

The canonical template suppresses the thinking channel at the start of a normal model turn, but emits nothing after a tool response when enable_thinking is false. The model may then open a thinking channel on its own, and a quantized model sometimes writes the literal word thought into the answer. This patch gives the tool_response branch the same suppression:

{%- elif ns.prev_message_type == 'tool_response' -%}
    {%- if enable_thinking -%}
        {{- '<|channel>thought\n' -}}
    {%- else -%}
        {{- '<|channel>thought\n<channel|>' -}}
    {%- endif -%}
{%- endif -%}

Only that case changes — the other prompt paths render byte-identical to the canonical template. To get stock behaviour, replace chat_template.jinja with the one from the base model repo; the weights are unaffected.