Gemma 4 31B IT — QAT q4_0, grid-faithful MLX (20 GB)

Built with Gemma. A grid-faithful MLX conversion of Google's QAT q4_0 checkpoint of Gemma 4 31B IT — the quantization grid the model was quantization-aware-trained for, emitted directly in MLX's native affine format. Runs on Apple Silicon via mlx-lm and MLX-Swift with stock kernels.

Why grid-faithful matters

Google QAT-trained these weights against the q4_0 grid (int4, group size 32, symmetric): ~53% of weights sit exactly on grid points. Re-quantizing with a generic asymmetric min-max quantizer builds a different grid and snaps those weights to the wrong points — we measured +6.3% deviation from the model Google validated. Instead, this conversion computes the q4_0 codes (d = absmax/−8, q = round(w/d)+8) and emits them as native MLX affine triplets (scales = d, biases = −8·d, gs32) — mathematically the same grid.

Measured fidelity (rel-RMS, sampled across all depths):

Reference	This artifact	Naive MLX min-max
q4_0-intended model (what Google validated)	0.135% (bf16-scale rounding only)	6.32%
BF16 QAT master	5.03% (≈ the 4.99% the QAT targets)	6.15%

Layout (mirrors Google's shipped GGUF / compressed-tensors exactly)

Tensor class	Precision
All decoder linears (q/k/v/o, gate/up/down, 60 layers)	4-bit gs32, q4_0-faithful
`embed_tokens` (tied lm_head)	6-bit gs64 (mirrors Q6_K)
Norms, QAT `layer_scalar`, rope	unquantized
Vision tower	bf16, unquantized (as Google ships it)

Benchmarks (Apple M5 Max, 40-core GPU, 614 GB/s)

Decode: ~26 tok/s single-stream (≈90% of the memory-bandwidth ceiling for a dense 31B at ~5 bpw)
Peak memory: 19.7 GB

Usage

pip install mlx-lm
python -m mlx_lm generate --model EigenLabs/gemma-4-31B-it-qat-q4_0-MLX \
  --prompt "Explain quantization-aware training in one paragraph."

Vision tensors are included with original names: text-only loaders skip them; mlx-vlm can consume them for multimodal use.

Provenance & terms

Converted from google/gemma-4-31B-it-qat-q4_0-unquantized with a deterministic, calibration-free converter (no further training). Weight values are bit-derived from Google's QAT release; use is subject to the Gemma Terms of Use. Fidelity numbers are weight-space; the conversion is designed to inherit the QAT checkpoint's validated quality by construction rather than re-evaluating it.