gemma-4-12B-it-qat-4bit (MLX)
Gemma 4 12B Unified instruction-tuned, quantization-aware-trained (QAT), converted to MLX 4-bit weights for Apple Silicon. This is a conversion of Google's own QAT checkpoint. It is not an official Google release.
- Source:
google/gemma-4-12B-it-qat-q4_0-unquantized, revisionb6ed86275a6a5735884e208bfed95b445a684ca2(Apache 2.0, not gated). - Architecture: Gemma4UnifiedForConditionalGeneration (gemma4_unified). Inputs are text, image and audio as in the source model; the vision and audio embedders are included. Only text and image input were exercised in the checks below.
- Size: 10.99 GB of weights (7.355 bits per weight on average).
- Converted: 2026-10-26 with mlx-vlm 0.7.2, mlx 0.32.2, transformers 5.17.0, Python 3.12.
How it was quantised
| Setting | Value |
|---|---|
| Mode | affine |
| Bits (default) | 4 |
| Group size | 64 |
| 8-bit (group size 64) | the dense MLP projections (mlp.gate_proj, mlp.up_proj, mlp.down_proj) in all 48 layers (144 modules) |
| Not quantised | normalisation weights, layer scalars and the vision positional embedding. The embedding table, embed_vision, embed_audio and vision_embedder.patch_dense are quantised at the default 4 bits |
| Weight dtype before quantisation | bfloat16 (the dtype of the source) |
The per-module overrides are recorded in config.json under quantization. tokenizer.json, chat_template.jinja and
generation_config.json are byte-identical to the source checkpoint's; config.json, tokenizer_config.json and
processor_config.json were re-saved by mlx-vlm and transformers.
Convert command
mlx_vlm.convert on its own cannot keep part of the model at 8 bits, so the conversion calls the same function
(mlx_vlm.convert.convert) from a short script. Run it with the downloaded Google checkpoint as the first argument.
It downloads nothing and runs with trust_remote_code=False.
"""Convert a Google Gemma 4 QAT (q4_0-unquantized) checkpoint to MLX 4-bit weights.
Usage (always run with -I, from a directory that is NOT the downloaded checkpoint):
python -I convert_gemma4_qat.py <local-snapshot-dir-of-google-checkpoint> <output-dir> [--compare-config <config.json>]
What it does: calls mlx_vlm.convert.convert() (the same function `python -m mlx_vlm.convert`
runs) on the CPU stream, with 4-bit affine, group size 64, bfloat16, and ONE thing the command line cannot say: a
quantisation predicate that keeps the dense MLP projections (gate_proj / up_proj / down_proj)
and the MoE router projection at 8 bits (group size 64). Everything else in the language model,
including the embedding table and the attention projections, is 4-bit. Vision / audio towers
are skipped by mlx-vlm's own skip list (`skip_multimodal_module`), exactly as `mlx_vlm.convert --quantize` does.
It does not download anything and runs with trust_remote_code=False.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
from mlx_vlm.utils import skip_multimodal_module
GROUP_SIZE = 64
BITS = 4
KEEP_8BIT = re.compile(
r"(^|\.)language_model\.model\.layers\.\d+\.(mlp\.(gate_proj|up_proj|down_proj)|router\.proj)$"
)
def predicate(path, module):
"""Per-module quantisation: 8-bit for the dense MLP and the router, base 4-bit otherwise.
mlx-vlm's own default predicate skips the vision / audio towers; passing a predicate replaces
that default, so the same skip list is applied here first.
"""
if not hasattr(module, "to_quantized"):
return False
if skip_multimodal_module(path):
return False
if KEEP_8BIT.search(path):
return {"group_size": GROUP_SIZE, "bits": 8}
return True
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("source")
ap.add_argument("output")
ap.add_argument("--compare-config", default=None,
help="a reference config.json whose `quantization` block the result must equal")
args = ap.parse_args()
import os # noqa: PLC0415
import shutil # noqa: PLC0415
# mlx-vlm copies the source's *.json files into the output with shutil.copy, which keeps the
# source's permission bits, then calls processor.save_pretrained() over them. A source that
# comes from the Hugging Face cache has tokenizer.json at mode 0444, so that second write is
# refused ("Permission denied"). Make every copied file writable; contents are untouched.
_copy = shutil.copy
def _copy_writable(src_file, dst, *a, **k):
result = _copy(src_file, dst, *a, **k)
os.chmod(result, 0o644)
return result
shutil.copy = _copy_writable
# Quantise on the CPU stream. The MLX GPU quantiser rounds a small share of weights (0.2 to 1.5
# percent of elements, by one step) differently from the CPU one; the CPU result is the one that
# is byte-identical to the existing mlx-community builds of these two models. It also keeps the
# 26B-A4B conversion from hitting the Metal command-buffer timeout on a 34 GB Mac, where the 50 GB
# source shard is memory-mapped and read lazily.
import mlx.core as mx # noqa: PLC0415
mx.set_default_device(mx.cpu)
from mlx_vlm.convert import convert # noqa: PLC0415
src, out = Path(args.source), Path(args.output)
if not (src / "config.json").is_file():
sys.exit(f"not a checkpoint directory: {src}")
if out.exists() and any(out.iterdir()):
sys.exit(f"output directory is not empty: {out}")
convert(
str(src),
mlx_path=str(out),
quantize=True,
q_group_size=GROUP_SIZE,
q_bits=BITS,
q_mode="affine",
dtype=None, # the checkpoint's own dtype (bfloat16)
trust_remote_code=False,
quant_predicate=predicate,
)
cfg = json.loads((out / "config.json").read_text(encoding="utf-8"))
quant = cfg["quantization"]
base = {k: v for k, v in quant.items() if not isinstance(v, dict)}
overrides = {k: v for k, v in quant.items() if isinstance(v, dict)}
print("RESULT base quantization:", base)
print("RESULT per-module overrides:", len(overrides),
sorted({(v['bits'], v['group_size']) for v in overrides.values()}))
if args.compare_config:
ref = json.loads(Path(args.compare_config).read_text(encoding="utf-8"))["quantization"]
same = ref == quant
print("RESULT quantization block equals the reference config:", same)
if not same:
only_ref = sorted(set(ref) - set(quant))
only_new = sorted(set(quant) - set(ref))
print(" only in reference:", only_ref[:10], len(only_ref))
print(" only in new:", only_new[:10], len(only_new))
return 3
return 0
if __name__ == "__main__":
raise SystemExit(main())
Checks
These were run on an Apple M-series Mac (32 GB), with the same prompts and greedy decoding (temperature 0) on this build and on
mlx-community/gemma-4-12B-it-qat-4bit (revision e70c6b3b), an existing MLX build of the same model.
- Tensor by tensor: all 1341 tensors (names, shapes, dtypes and every byte of the values) are identical to
mlx-community/gemma-4-12B-it-qat-4bitat revisione70c6b3b, and thequantizationblock ofconfig.jsonis identical. That repository's card names no licence and no base model; this build reproduces its weights from Google's published Apache-2.0 checkpoint and documents where they come from. - Loads and runs with
mlx-vlm0.7.2 (mlx_vlm.load,generate, chat template, image input). Active GPU memory after load: 10.99 GB. - Greedy decoding, 10 text prompts of up to 160 new tokens each: 10 of 10 are token-for-token identical.
- Typed decisions (one forward pass, the option-letter logits), 12 questions: the top option is the same in 12 of 12; largest difference in any option probability 0.0e+00; largest KL divergence of the full next-token distribution 0.0e+00 nats.
- Image input, 6 typed questions on two test pictures: the top option is the same in 6 of 6.
Use
pip install -U mlx-vlm
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
repo = "astrocyte-productions/gemma-4-12B-it-qat-4bit"
revision = "<commit id>" # pin one; do not follow main in production
model, processor = load(repo, revision=revision)
config = load_config(repo, revision=revision)
prompt = apply_chat_template(processor, config, "Say hello in one sentence.", num_images=0)
result = generate(model, processor, prompt, max_tokens=64, temperature=0.0, verbose=False)
print(getattr(result, "text", result))
Licence
Apache License 2.0, the licence of the source checkpoint (Google publishes it at
https://ai.google.dev/gemma/docs/gemma_4_license). The full text is in LICENSE; it is the standard Apache 2.0 text,
and its terms match the text on that page.
Google's Gemma 4 QAT repositories contain no LICENSE or NOTICE file of their own, so no Google NOTICE is included here.
The original model is by Google DeepMind. This conversion changes only the numeric format of the weights
(quantisation) and adds the MLX metadata in config.json. It comes with no warranty.
Intended use, limitations and safety information are those of the source model; see the Google model card.