Gemma4 26B-A4B QAT Uncensored (HauhauCS Balanced) with MTP and vision, NInfer v3 artifact

A NInfer v3 artifact of HauhauCS's Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP (Gemma 4 26B-A4B it QAT with refusals removed, the Balanced variant, Q4_K_M) together with the drafter and the vision projector the release ships, for NInfer-all, the master branch of iamwavecut/ninfer-all. Every matrix of the model keeps the release's ggml blocks byte for byte, each in its own format, so nothing is requantized; the engine multiplies the blocks in place, drafts with the release's MTP head (--spec mtp) and encodes images and video with the vision tower (--vision). Stock NInfer builds refuse this file: the Gemma 4 family and the gguf_* formats exist only in that line, the vision tower from commit 877fe16e on. The family was developed and measured on the RTX 3090; the RTX 40 and RTX 50 builds compile it, unmeasured.

It runs the whole model: images and video beside text, one to eight concurrent requests with prompt-prefix reuse (pictures included), structured output, tool calls in Gemma's markup returned as OpenAI tool calls, and reasoning in its own channel when a request turns thinking on.

What is inside

One .ninfer file of 18,194,661,888 bytes (16.95 GiB; SHA-256 784bce41fc80481da6e63ba42aff97e165f9fd192845cdbcd83a16b4aadaf383), next to its conversion report, SHA256SUMS and NOTICE: 1,061 parameters in Q4_K (133 objects), Q8_0 (28), Q5_0 (32), Q4_0 (19, the drafter), Q6_K (14), BF16 (111, the vision tower) and FP32 (580).

Conversion command, from the repository's tree at 877fe16e, with the release at revision f9093662 (Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf, sha256 3c131334…; mtp-gemma-4-26B-A4B-it.gguf, sha256 62bd3af7…; mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf, sha256 b5346e5b…) and Google's text files at 4d7ae498:

mkdir -p hf
for f in config.json tokenizer.json tokenizer_config.json chat_template.jinja generation_config.json \
         processor_config.json; do
  curl -fsSL -o hf/$f https://huggingface.co/google/gemma-4-26B-A4B-it/resolve/main/$f
done
hf download HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP \
  Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf mtp-gemma-4-26B-A4B-it.gguf \
  mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf --local-dir .
# The release's sampling as the artifact's defaults.
jq '. + {do_sample: true, temperature: 0.6, top_k: 64, top_p: 0.9, min_p: 0.05}' \
  hf/generation_config.json > generation_config.json
python3 -m tools.convert --model hf --recipe gemma4_gguf --components text,mtp,vision \
  --source gguf=Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf \
  --source mtp=mtp-gemma-4-26B-A4B-it.gguf \
  --source vision=mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf \
  --resource generation_config.json=generation_config.json \
  --name Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP \
  --out Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP-ninfer-v3.ninfer --device cpu

The conversion takes about a minute.

Running

Four concurrent requests over 40K-token contexts with two drafts a round, images and video included, from the Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit; images from sha-877fe16eff19 on read this file):

hf download WaveCut/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP-NInfer-v3 \
  Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP-ninfer-v3.ninfer \
  --model-id gemma4-hauhau --max-context 40960 --max-concurrency 4 --spec mtp --draft-tokens 2 --vision

The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer) takes the same flags:

ninfer-serve models/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP-ninfer-v3.ninfer \
  --model-id gemma4-hauhau --max-context 40960 --max-concurrency 4 --spec mtp --draft-tokens 2 --vision

The weights take 15.6 GiB on the GPU (0.25 GiB more for the drafter with --spec mtp) and the prompt workspace about 1.1 GiB; a sequence holds 250 MiB for the sliding windows with --spec mtp (200 MiB without) and 10 KiB per token of context for the full-attention blocks. The release card describes the model as reasoning before it answers; NInfer follows the chat template, where thinking is off unless a request asks (chat_template_kwargs: {"enable_thinking": true}).

Images and video. With --vision the requests take pictures and videos (OpenAI image_url / video_url parts, Anthropic image blocks; data URIs, URLs or, in the CLI, files). Each picture, and each of 32 frames sampled evenly from a video, becomes soft tokens as Google's processor lays them out: an image takes 280 (its detail low 70, high the --vision-max-merged bound, 1120 by default), a video frame 70, each picture resized with its aspect kept. --vision-residency places the tower: resident on the GPU (1.07 GiB), overlay in pinned host memory streamed through a two-block GPU window (0.14 GiB on the GPU), cpu as an FP32 copy on CPU threads (none on the GPU; its unit bound defaults to 140 soft tokens).

Speculative decoding. The drafter proposes tokens that the model verifies in one pass, keeping a draft only while it equals the request's own sample: the output is the one plain decoding gives, and the model's behaviour is unchanged by the drafter. A request decoding alone takes the configured drafts, two or three at once one draft each, four or more decode plainly.

Quality

The conversion keeps every block of the release byte for byte; the same build's kernels are checked against FP64 oracles and its assistant path against an FP64 transcription of transformers' Gemma 4 drafting, and its vision tower against an FP64 tower over the same mmproj tensors (2.2% relative RMS error on the GPU, 0.17% on CPU threads; see the repository's Gemma 4 notes). Perplexity over eleven 8,192-token windows of a NInfer source text: 21.79 (Google's QAT Q4_0 release on the same windows: 24.79). Before publication this file served greedy chat requests (code, a Russian answer, a 13,600-token prompt) plainly and with two drafts (acceptance 0.63), image and video requests (a picture's text and contents, two pictures compared, the events of a video in order, a follow-up turn over the same picture), and the CLI. Reasoning benchmarks were not run.

Speed

October 2026, one RTX 3090 (24 GB) at 350 W, CUDA 12.8, a build of master at 03bd4006: prefill 6,113 tok/s for a 2,048-token prompt and 5,623 for 10,000; decode 206.2 tok/s from an empty context and 182.7 after 10,000 tokens. Ten chat requests (code, explanation, summary, QA, translation, math, a story, a Russian answer, a 13,600-token source excerpt), 384 greedy tokens each, one at a time, decode tok/s:

Vision encoding on the GPU, after the server's warm-up: a 768 x 512 picture (260 soft tokens) in 71 ms, the same at detail: high (1,080) in 412 ms, a 32-frame video (32 x 66) in 413 ms.

Credits and license

Uncensored weights, quantization, drafter and mmproj GGUFs: HauhauCS, whose release declares the Gemma license (license: gemma), kept here; the drafter, per the release, comes from Unsloth's Gemma 4 GGUFs. Base model, vision tower, tokenizer, chat template and processor configuration: Google, Gemma 4, under the Gemma 4 terms (Apache 2.0). Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.