Gemma 4 12B — 23% lower KL than Google's official W4A16 QAT

Native QUASAR W4A16-G64 checkpoint for vLLM: 0.034 vs 0.044 KL to BF16, higher point estimates on every reported downstream benchmark, and 4.25 vs 4.5 quantized bpw, versus Google's official W4A16 QAT.

Teacher-forced against BF16 google/gemma-4-12B-it on 512 held-out chat prompts, thinking on; same prompts and BF16 reference for both checkpoints.

The GGUF twin independently shows the same result under llama.cpp: 0.048 vs 0.069 KL to BF16 (30% lower) versus Google's Q4_0 QAT.

vllm serve QUASAR-QAT/gemma-4-12B-it-QUASAR-W4A16-G64 --dtype float16 --limit-mm-per-prompt '{"image":0}'

Standard compressed-tensors W4A16; 262K context. For llama.cpp / Ollama / LM Studio, use the GGUF twin.

Downstream benchmarks

Higher point estimates on all four downstream benchmarks than Google's QAT model.

Same harness and backend for both checkpoints; the MMLU gain is +2.6 points (95% CI [+2.0, +3.2], paired bootstrap over 14,042 questions). Full configs, BF16 results and CIs are in EVAL.md.

Native QAT checkpoint

Native QAT checkpoint: the INT4 codes come directly from training, rather than a post-training re-quantization. export_receipt.json verifies the export per tensor (328 quantized linears). The GGUF twin carries the same codes and scales and ships a --verify script that checks the two files against each other.

Run it

• vLLM: vllm serve QUASAR-QAT/gemma-4-12B-it-QUASAR-W4A16-G64 --dtype float16 --limit-mm-per-prompt '{"image":0}', then any OpenAI-compatible client. Text serving tested on vLLM 0.28.0 and 0.29.0 (--dtype float16 keeps the stored fp16 scales exact); 262,144-token context. For vision, use the GGUF twin with its projector.
• Format: 328 decoder linears int4 symmetric, group_size 64, fp16 scales, compressed-tensors pack-quantized; everything else bf16. Full-attention layers share K/V (attention_k_eq_v), exported exactly as the model defines. 7.9 GB.

QUASAR

QUASAR is loss-aware quantization-aware training. It improves the low-bit reconstruction during training, then exports to standard deployment formats with zero inference overhead.

This checkpoint was healed against the BF16 teacher for ~600M tokens: one epoch over ~377k prompts self-distilled from BF16 Gemma 4 12B-it with thinking on, fp32 master weights, INT4 group-64.

Related

• GGUF (llama.cpp / Ollama / LM Studio) twin of this checkpoint: QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF
• Gemma 4 E4B, same recipe: GGUF · W4A16 · collection · Gemma 4 E4B & 12B — QUASAR 4-bit QAT
• Base model: google/gemma-4-12B-it · Paper: arXiv:2608.13966 · Org: huggingface.co/QUASAR-QAT

License

Based on Gemma 4 by Google DeepMind. Upstream LICENSE/NOTICE apply.

Citation

arxiv.org/abs/2608.13966

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}