Gemma 4 26B-A4B IT Assistant Q6_K GGUF

This repository contains a Q6_K GGUF quantization of the Gemma 4 26B-A4B assistant head for use as an MTP draft model in ik_llama.cpp.

It uses the native ik_llama Gemma 4 MTP schema:

general.architecture = gemma4_mtp

This is not intended for stock llama.cpp, which does not currently support Gemma 4 MTP the same way as ik_llama.cpp.

Related quant

There is also a smaller Q4_K_S sibling quant here: ji-farthing/gemma-4-26B-A4B-it-assistant-Q4_K_S-GGUF.

The Q6_K file is about 37 MiB larger than the Q4_K_S file in this local build. Try Q4_K_S first if you are tight on VRAM; try Q6_K if you want the slightly heavier assistant head.

File

How this was made

Source file:

• gemma-4-26B-A4B-it-assistant-F16.gguf
• Source repository: Radamanthys11/Gemma-4-26B-A4B-it-assistant-GGUF
• Upstream base model: google/gemma-4-26B-A4B-it-assistant

Quantized with ik_llama.cpp:

• Commit: 397150ca MTP: faster recurrent state restore (#1791)
• llama-quantize SHA256: 15ca21cbd3881326cec391325d663fed94122fe87a42d84373682b1612d9dc52
• No imatrix was used.
• This was quantized from F16, not requantized from Q8.

Command:

/home/joel/Projects/ik_llama.cpp/build-cuda/bin/llama-quantize \
  gemma-4-26B-A4B-it-assistant-F16.gguf \
  gemma-4-26B-A4B-it-assistant-Q6_K.gguf \
  Q6_K \
  12

Quantizer output reported:

main: build = 4482 (5fe010f0)
llama_model_quantize_internal: model size  =   800.57 MB
llama_model_quantize_internal: quant size  =   328.40 MB

Usage

Use this as the draft model alongside a Gemma 4 26B-A4B target GGUF:

./build/bin/llama-server \
  -m gemma-4-26B-A4B-it-UD-IQ4_XS.gguf \
  -c 20480 \
  --model-draft gemma-4-26B-A4B-it-assistant-Q6_K.gguf \
  --spec-type mtp \
  -ngld 999 \
  --draft-max 2 \
  --draft-p-min 0.0 \
  -ctk q4_0 \
  -ctv q4_0 \
  --jinja

--draft-max 1 and --draft-max 2 are both reasonable starting points. Local results are workload-sensitive, so treat this as a practical assistant option rather than a universal speed claim.

Local smoke test

Smoke-tested locally with:

• Target: gemma-4-26B-A4B-it-UD-IQ4_XS.gguf
• Draft: this Q6_K assistant
• ik_llama.cpp commit: 397150ca
• Context: 20480
• --draft-max 2

On a short local chat-completions smoke against the Q4_K_S native assistant, the Q6_K head was larger but steadier:

This is only a tiny local check on one RTX 4070 setup, but it suggests Q6_K may be a useful middle point between the smaller Q4_K_S head and the larger public Q8_0 head.

License

This quantization follows the Gemma license inherited from the upstream Google Gemma model.