Gemma 4 26B A4B Heretic W4A16 G64 RTN for vLLM
compressed-tensors W4A16 quantization of
WaveCut/gemma-4-26B-A4B-it-heretic
for vLLM on a single 24 GB RTX 3090.
Quantization
| Setting | Value |
|---|---|
| Method | RTN / memoryless_minmax |
| Weights | signed INT4, symmetric |
| Activations | BF16 at runtime |
| Group size | 64 |
| Format | compressed-tensors, pack-quantized |
| Source architecture | Gemma4ForConditionalGeneration |
| Source Heretic commit | e6d09d5c434a5e91326ce9d779786a91e7aaae35 |
| Original quantization stack | llm-compressor main, Transformers 5.8.0 |
Vision modules, routers, embeddings, and lm_head remain unquantized. Group
size 64 is required because Gemma 4 projection widths such as 2112 and 704 are
not divisible by 128. The exact recipe is stored in recipe.yaml.
Source refresh
On 2026-07-16, Google updated the Gemma 4 chat template but did not update the
model weights. The initial and current Google revisions have identical LFS
SHA256 values for both BF16 weight shards. This repository therefore keeps the
existing verified W4A16 tensors and refreshes the chat template, tokenizer
metadata, and processor metadata from Google revision
01e5b3ee840d3a9e0b0b493c593e85398a30ef75.
vLLM
The production profile uses:
vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
--revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \
--served-model-name "Gemma 4 26B Heretic" \
--trust-remote-code \
--dtype bfloat16 \
--max-model-len 40192 \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.95 \
--generation-config vllm \
--no-enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4
The checkpoint was accepted for production on AI Farm with vLLM 0.25.1, Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0. On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case groups plus streaming checks. The matrix included exact needle recovery at 39,186 text tokens and 38,490 multimodal tokens, including concurrency 2. Both runtimes also passed 24/24 production-shaped Russian quality probes; median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1.
The full environment, timing table, harness, and raw JSONL artifacts are in
the AI Farm vLLM report.
The historical RTX 3090 quantization benchmark and harness remain available
under bench/ and scripts/.
Limitations
- RTN is a weight-only quantization; compare output quality against the BF16 Heretic checkpoint for quality-sensitive workloads.
- The 40,192-token, two-request profile is a tight fit on 24 GB and leaves little free VRAM.
- This is a third-party post-processed and quantized checkpoint. It is not affiliated with or endorsed by Google.