Gemma 4 26B A4B-it QAT assistant (drafter), exact Q4_0 GGUF (unofficial)

This is an unofficial build, made and published independently of Google. It is a llama.cpp GGUF of Google's speculative-decoding drafter for Gemma 4 26B A4B-it QAT, google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant, with every weight matrix stored at 4 bits on the grid quantization-aware training produced. It is 251,939,648 B, against 461,785,920 B for the Q8_0 file llama.cpp's converter writes. Google's model card is kept unchanged in ORIGINAL_README.md.

Use it with a Gemma 4 26B A4B-it QAT target, for example google/gemma-4-26B-A4B-it-qat-q4_0-gguf. Google's card says the drafter must match the target's QAT precision.

The 26B target is 14.4 GB, so on an 8 GB card its experts go to system RAM with -ncmoe (the experts of the first N of 30 layers stay on the CPU). The setting this was measured at:

llama-server -m gemma-4-26B_q4_0-it.gguf -ngl 99 -ncmoe 21 -fa 1 -c 131072 -ctk q4_0 -ctv q4_0 -t 8 \
  -md gemma-4-26B-A4B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2

Measured on llama.cpp build 7182ef71e only.

Without a drafter the target runs at 29.6 tok/s. This file is 1.03x the Q8_0 one at the same acceptance.

Per prompt at 2 drafted tokens: code 1.51x, facts 1.43x, explanation 1.17x, chat 1.16x, story 1.12x, summary 1.12x.

The gain is smaller than on the dense 12B (2.01x for its drafter on the same card). With the experts on the CPU, each drafted token the target checks routes to its own 8 of 128 experts, which the CPU then reads; that likely limits the gain here (inferred, not measured). Use 1 or 2 drafted tokens, not 3.

Measured on an RTX 3050 OEM (8 GB) in an i7-5820K desktop (6 cores, quad-channel DDR4), llama.cpp 7182ef71e, target google/gemma-4-26B-A4B-it-qat-q4_0-gguf with -ncmoe 20 -t 8, flash attention, f16 KV cache, -c 8192: six prompts (code, chat, story, explanation, facts, summary), 256 tokens, greedy, median of three runs, geometric mean over prompts. No drafter ran first and last, at 29.6 and 28.7 tok/s.

On an 8 GB card the drafter costs context. With a q4_0 KV cache, the target loads at 131,072 tokens with 21 expert layers in RAM and this drafter; without a drafter, 262,144 fits with 22 in RAM. 262,144 does not load with the drafter even at 24.

Greedy text changes. With the drafter, greedy replies differ from the target's greedy replies without a drafter in 18 of 18 at 2 drafted tokens (12 of 18 at 1). The target still checks every drafted token; the differences likely come from rounding in the batched check (inferred, not measured). Answer quality with this drafter has not been measured on the 26B; for the 12B's drafter, built the same way, GSM8K and BFCL scored level with and without it.

What is in it

• Built from google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant at revision 9537141: 4 transformer layers of width 1024 (3 sliding-window, 1 full attention), with queries only; keys and values come from the target's cache. Two projections join it to the target's 2816-wide hidden state. The output head is the tied 262,144-token embedding. The drafter itself is dense; only the target is a mixture of experts.
• Every weight matrix as Q4_0 (the embedding, 4 x attention q/o, 4 x MLP gate/up/down, both projections), each block's trained step recovered: largest magnitude divided by 8, 7, … 1, whichever puts all 32 values on an integer level, refined by least squares and stored as the fp16 block scale. No block was off the grid. 96.6–97.0% of values per tensor reproduce the bf16 source exactly once rounded back to bf16.
• Metadata (hyperparameters, tokenizer, chat template) is llama.cpp's converter output, general.file_type set to Q4_0. Norms and scales stay F32.
• Text only. The drafter has no vision or audio parts.

Files

evidence/ holds the build script (gguf_exact.py), its per-tensor log and report (build.log, build-report.json), the speed run's script and per-prompt table (spec_ab.py, speed-summary.txt) and the converter log. The full run is in local-llamacpp-3050oem-26b-q4_0.

License

Apache 2.0, as Google's source. Gemma 4 terms: https://ai.google.dev/gemma/docs/gemma_4_license