Gemma 4 12B IT QAT — GGUF

GGUF quantizations of google/gemma-4-12B-it-qat-q4-0, created from the Quantization-Aware Training (QAT) checkpoint of Gemma 4 12B IT.

Model Overview

Gemma 4 12B IT is a multimodal model built by Google DeepMind that handles text, image, and audio inputs and generates text output. It is designed for efficient on-device and server deployment.

This repository contains GGUF conversions of the QAT-optimized checkpoint, making it usable with llama.cpp and other GGUF-compatible inference engines.

Model Architecture

GGUF Files

A chat_template.jinja file is also provided for use with chat-based inference.

Note on QAT: The Q4_0 file is the native QAT quantization. The K-quant and Q8_0 variants are additional GGUF quantizations produced from the QAT checkpoint. The QAT optimization preserves quality close to bfloat16 while dramatically reducing memory requirements.

Note on mmproj: The mmproj.gguf file contains the vision and audio projectors needed for multimodal (image/audio) inference. It is shared across all quantization variants.

Usage

llama.cpp (CLI)

# Run text-only inference
./llama-cli \
  -m gemma-4-12b-it-qat-Q4_0.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-k 64 --top-p 0.95

llama-server (OpenAI-compatible API)

# Text-only
./llama-server \
  -m gemma-4-12b-it-qat-Q4_0.gguf \
  --host 0.0.0.0 --port 8080

# Multimodal (image + audio)
./llama-server \
  -m gemma-4-12b-it-qat-Q4_0.gguf \
  --mmproj mmproj.gguf \
  --host 0.0.0.0 --port 8080

Multimodal (Image / Audio)

For image and audio inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing image/audio data alongside text prompts.

Modality order tip: For best results, place image content before text and audio content after text in your prompt.

Generation Parameters

Recommended parameters from the model's generation_config.json:

Thinking Mode

Gemma 4 supports configurable thinking (reasoning) mode:

• Enable: Include the <|think|> token at the start of the system prompt.
• Output format: When thinking is enabled, the model outputs internal reasoning followed by the final answer:

  <|channel>thought
  [Internal reasoning]
  <channel|>
  [Final answer]
  

• Disable: Omit the <|think|> token.

Many libraries like Transformers and llama.cpp handle the chat template complexities automatically.

Key Features

• Multimodal: Text, image, and audio understanding
• Long Context: 256K token context window
• Function Calling: Native support for structured tool use (agentic workflows)
• Multilingual: Support for 140+ languages
• Native System Prompt: Supports the system role for structured conversations

Acknowledgements

• Original model: google/gemma-4-12B-it
• QAT checkpoint: google/gemma-4-12B-it-qat-q4-0
• Technical report: Gemma 4 Technical Report (arXiv:2607.02770)

Citation

@misc{gemmateam2026gemma4,
      title={Gemma 4 Technical Report},
      author={Gemma Team},
      year={2026},
      eprint={2607.02770},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.02770},
}

License

Apache License 2.0