Gemma 4 26B-A4B-it QAT, compressed-tensors W4A16, text only (unofficial repack)
This is an unofficial repack, made and published independently of Google. It is the language model of the repack of google/gemma-4-26B-A4B-it-qat-q4_0-unquantized, xbill9/gemma-4-26B-A4B-it-qat-q4_0-w4a16-ct, with the vision and audio towers removed, loadable as a text-only Gemma4ForCausalLM. Google's model card is kept unchanged in ORIGINAL_README.md.
It stores Google's quantization-aware-trained weights exactly as QAT left them on the Q4_0 grid. Google's own -qat-w4a16-ct export instead re-quantizes them with a min-max round-to-nearest quantizer (scale max|w| / 7.5), which moves every weight onto a shifted grid, and it also stores a bf16 lm_head byte-identical to the tied token embedding.
What it is
- Dropped: every tensor under
model.vision_tower.andmodel.embed_vision.(356 tensors, 1.06 GiB). - Kept: 35,567 language-model tensors, names unchanged (vLLM's
Gemma4ForCausalLMmapsmodel.language_model.*itself). - Verified with
repack_q4_0.pyverifyagainst Google's source: 765,107,200 groups of 32, none off the source grid; every unquantized tensor byte-identical. - Config: the source
text_configwitharchitectures: ["Gemma4ForCausalLM"]and thequantization_config, the tower modules removed from itsignorelist.processor_config.jsonis not included. Made withtext_only.py. - Size: 14.23 GiB, against 15.29 GiB with the towers.
Measured
Measured: suite 75.4% on one TPU v5e chip through the same text-only class (with loader changes to fit 15.75 GiB), and 75.3% on one v6e chip as the multimodal repack, whose language model this is; bf16 scores 76.4% on a v6e-4. Logs and per-record outputs: https://github.com/xbill9/gemma4-dev (jev-tpu-v5e1/, jev-tpu-31b/).
Serving it
No --limit-mm-per-prompt or --hf-overrides flag is needed; there is no multimodal path.
Google Cloud TPU. vLLM's TPU backend reads hf_config.text_config, which a text-only config does not have; it needs the one-line fallback in patches/textonly.diff and patches/lowmem.diff, plus tpu-inference #3653 for W4A16. This text-only config has not yet been served on TPU.
NVIDIA GPUs. Untested with this checkpoint. The E2B build made the same way (xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text) serves on one T4 with vLLM 0.29.0.
Limitations
- Text only: image and audio inputs are not supported.
- Unofficial. Report problems here, not to Google.
License and attribution
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed container format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.