Gemma 4 26B-A4B it QAT Q4_0 with MTP assistant and vision, NInfer v3 artifact

A NInfer v3 artifact of Google's quantization-aware-trained Gemma 4 26B-A4B it together with its QAT multi-token-prediction assistant and its vision tower, for NInfer-all, the master branch of iamwavecut/ninfer-all. Every matrix of the model keeps the release's ggml blocks byte for byte, so nothing is requantized; the engine multiplies the blocks in place, drafts with the assistant (--spec mtp) and encodes images and video with the tower (--vision). Stock NInfer builds refuse this file: the Gemma 4 family and the gguf_* formats exist only in that line, the vision tower from commit 877fe16e on. The family was developed and measured on the RTX 3090; the RTX 40 and RTX 50 builds compile it, unmeasured.

It runs the whole model: images and video beside text, one to eight concurrent requests with prompt-prefix reuse (pictures included), structured output, tool calls in Gemma's markup returned as OpenAI tool calls, and reasoning in its own channel when a request turns thinking on.

What is inside

One .ninfer file of 15,838,003,712 bytes (14.75 GiB; SHA-256 cc84eea7531a6ac2117585fd3a96ef13b773f3bc67ff1d85d30da122302519ff), next to its conversion report, SHA256SUMS and NOTICE: 1,061 parameters in Q4_0 (199 objects), Q6_K (1), BF16 (111) and FP32 (580).

Conversion command, from the repository's tree at 877fe16e, with the QAT GGUF repository at revision d1c082be (gemma-4-26B_q4_0-it.gguf, sha256 3eca3b8f…; gemma-4-26B-it-mmproj.gguf, sha256 a359953a…), Unsloth's GGUF at 7b92b5b2 (mtp-gemma-4-26B-A4B-it.gguf, sha256 7272d975…) and the Hugging Face text files at 4d7ae498:

mkdir -p hf
for f in config.json tokenizer.json tokenizer_config.json chat_template.jinja generation_config.json \
         processor_config.json; do
  curl -fsSL -o hf/$f https://huggingface.co/google/gemma-4-26B-A4B-it/resolve/main/$f
done
hf download google/gemma-4-26B-A4B-it-qat-q4_0-gguf gemma-4-26B_q4_0-it.gguf gemma-4-26B-it-mmproj.gguf \
  --local-dir .
hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF mtp-gemma-4-26B-A4B-it.gguf --local-dir .
python3 -m tools.convert --model hf --recipe gemma4_gguf --components text,mtp,vision \
  --source gguf=gemma-4-26B_q4_0-it.gguf --source mtp=mtp-gemma-4-26B-A4B-it.gguf \
  --source vision=gemma-4-26B-it-mmproj.gguf --name Gemma-4-26B-A4B-it-QAT-Q4_0-MTP \
  --out gemma-4-26B-A4B-it-QAT-Q4_0-MTP-ninfer-v3.ninfer --device cpu

The conversion takes about a minute.

Running

Four concurrent requests over 40K-token contexts with two drafts a round, images and video included, from the Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit; images from sha-877fe16eff19 on read the vision tower):

hf download WaveCut/gemma-4-26B-A4B-it-QAT-Q4_0-MTP-NInfer-v3 gemma-4-26B-A4B-it-QAT-Q4_0-MTP-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/gemma-4-26B-A4B-it-QAT-Q4_0-MTP-ninfer-v3.ninfer \
  --model-id gemma4 --max-context 40960 --max-concurrency 4 --spec mtp --draft-tokens 2 --vision

The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer) takes the same flags:

ninfer-serve models/gemma-4-26B-A4B-it-QAT-Q4_0-MTP-ninfer-v3.ninfer --model-id gemma4 \
  --max-context 40960 --max-concurrency 4 --spec mtp --draft-tokens 2 --vision

# One prompt from the CLI; without --spec the assistant is not loaded, without --vision the tower.
ninfer models/gemma-4-26B-A4B-it-QAT-Q4_0-MTP-ninfer-v3.ninfer --prompt "Explain RoPE in two sentences."

The weights take 13.4 GiB on the GPU (0.25 GiB more for the assistant with --spec mtp) and the prompt workspace about 1.1 GiB. A sequence holds 250 MiB for the sliding windows with --spec mtp (200 MiB without) and 10 KiB per token of context for the full-attention blocks, which keep their values only: their keys equal the values times the key norm, rotated, and are derived as they are read. Thinking is off unless a request asks (chat_template_kwargs: {"enable_thinking": true}).

Images and video. With --vision the requests take pictures and videos (OpenAI image_url / video_url parts, Anthropic image blocks; data URIs, URLs or, in the CLI, files). Each picture, and each of 32 frames sampled evenly from a video, becomes soft tokens as Google's processor lays them out: an image takes 280 (its detail low 70, high the --vision-max-merged bound, 1120 by default), a video frame 70, each picture resized with its aspect kept. --vision-residency places the tower: resident on the GPU (1.07 GiB), overlay in pinned host memory streamed through a two-block GPU window (0.14 GiB on the GPU), cpu as an FP32 copy on CPU threads (none on the GPU; its unit bound defaults to 140 soft tokens).

Speculative decoding. The assistant drafts from the next token and the model's final hidden state, attending over the model's own KV cache; the model verifies the drafts in one pass whose logits are bit for bit those of plain decoding and keeps a draft only while it equals the request's own sample, so the output is the one plain decoding gives, greedy or sampled. A request decoding alone takes the configured drafts, two or three at once one draft each, four or more decode plainly (drafting measured slower there).

Quality

The assistant's drafts equal an FP64 transcription of transformers' Gemma 4 drafting (tools/reference/gemma4.py --assistant) on a 40-token prompt, and its verification logits equal the decoded ones bit for bit. The vision tower's soft tokens agree with an FP64 NumPy tower over the mmproj (tools/reference/gemma4_vision.py, itself equal to transformers' Gemma4VisionModel within FP64 rounding) to 2.2% relative RMS error with every token's cosine at least 0.988 on the GPU (transformers' own tower in BF16: 6.9% and 0.889), and to 0.17% on CPU threads. The kernels the model runs (GGUF vector and matrix products at Gemma's widths, the routed-expert products, decode attention with derived keys) are checked against FP64 oracles; see the repository's Gemma 4 notes. Perplexity over eleven 8,192-token windows of a NInfer source text: 24.79. Before publication this file served greedy chat requests (code, a Russian answer, a 13,600-token prompt) plainly and with two drafts (acceptance 0.65), image and video requests with each vision residency (a picture's text and contents, two pictures compared, the events of a video in order, a follow-up turn over the same picture), and the CLI. Reasoning benchmarks were not run.

Speed

October 2026, one RTX 3090 (24 GB) at 350 W, CUDA 12.8, a build of master at 03bd4006, against llama.cpp 23b0202 with the same GGUF on the same card:

Ten chat requests (code, explanation, summary, QA, translation, math, a story, a Russian answer, a 13,600-token source excerpt), 384 tokens each, one at a time, decode tok/s:

llama.cpp's llama-server with the same GGUF and drafter decodes the same greedy requests at 131.4 tok/s plainly and 201.6 with two drafts. Parallel chat streams, tok/s together, plain → with the assistant: 1 stream 208.4 → 295.5, 2 streams 310.1 → 370.0, 3 streams 354.7 → 365.7; 4 streams 397.2 decode plainly.

Vision encoding (a request's vision time, after the server's warm-up), the cpu residency on the same host's 4-core i3-12100F:

Credits and license

Model, QAT quantization, vision tower and its mmproj GGUF, QAT assistant, tokenizer, chat template and processor configuration: Google, Gemma 4, under the Gemma 4 terms (Apache 2.0). The assistant's Q4_0 GGUF: Unsloth. Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.