Gemma 4 26B-A4B IT Assistant Q4_K_S GGUF

This repository contains a Q4_K_S GGUF quantization of the Gemma 4 26B-A4B assistant head for use as an MTP draft model in ik_llama.cpp.

It is intended as a smaller native ik_llama.cpp assistant option for VRAM-constrained local setups. It uses the native ik_llama Gemma 4 MTP schema:

general.architecture = gemma4_mtp
general.file_type    = 14

This is not intended for stock llama.cpp, which does not currently support Gemma 4 MTP the same way as ik_llama.cpp.

Related quant

There is also a Q6_K sibling quant here: ji-farthing/gemma-4-26B-A4B-it-assistant-Q6_K-GGUF.

The Q4_K_S file is the smaller option. The Q6_K file is about 37 MiB larger in this local build and may be worth trying if you have the VRAM headroom.

File

How this was made

Source file:

• gemma-4-26B-A4B-it-assistant-F16.gguf
• Source repository: Radamanthys11/Gemma-4-26B-A4B-it-assistant-GGUF
• Upstream base model: google/gemma-4-26B-A4B-it-assistant

Quantized with ik_llama.cpp:

• Commit: f9a93c37 Fix GLM-4.5 MTP loading (#1784)
• llama-quantize SHA256: f4a622096e40ce496f3409516ee258227ac2d9f3b13cac0c0ba6d5d73e2c032b
• No imatrix was used.
• This was quantized from F16, not requantized from Q8.

Command:

/home/joel/Projects/ik_llama.cpp/build-cuda/bin/llama-quantize \
  gemma-4-26B-A4B-it-assistant-F16.gguf \
  gemma-4-26B-A4B-it-assistant-Q4_K_S.gguf \
  Q4_K_S \
  12

Quantizer output reported:

main: build = 4486 (f9a93c37)
llama_model_quantize_internal: model size  =   800.57 MB
llama_model_quantize_internal: quant size  =   291.21 MB

Usage

Use this as the draft model alongside a Gemma 4 26B-A4B target GGUF:

./build/bin/llama-server \
  -m gemma-4-26B-A4B-it-UD-IQ4_XS.gguf \
  -c 20480 \
  --model-draft gemma-4-26B-A4B-it-assistant-Q4_K_S.gguf \
  --spec-type mtp \
  -ngld 999 \
  --draft-max 2 \
  --draft-p-min 0.0 \
  -ctk q4_0 \
  -ctv q4_0 \
  --jinja

--draft-max 1 and --draft-max 2 are both reasonable starting points. On my RTX 4070 setup, --draft-max 2 was slightly faster on a small AutoQuant-like prompt, but this is workload-sensitive.

Local smoke test

Smoke-tested locally with:

• Target: gemma-4-26B-A4B-it-UD-IQ4_XS.gguf
• Draft: this Q4_K_S assistant
• ik_llama.cpp commit: f9a93c37
• Context: 20480

The server loaded the target as gemma4, loaded this assistant as gemma4_mtp, and reached:

common_speculative_state_mtp: MTP context ready

In a tiny local comparison against Sam/Radamanthys Q8_0, the Q8 head was still a little faster, while this Q4_K_S head used less disk/VRAM and remained viable. This repo is therefore best understood as a smaller compatibility/convenience option, not as a claim of better performance than Q8.

License

This quantization follows the Gemma license inherited from the upstream Google Gemma model.