bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF

We recommend ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf over Q4_K_M at this size: it is the file evaluated below, and it held up better on our math benchmarks. Pull it with -hf bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF:MXFP4_MOE.

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-gemma-4-26B-A4B-it — ThinkingCap's shorter-thinking post-training applied to gemma-4-26B-A4B-it, a mixture-of-experts model with image input.

➡️ Full model description, evaluation results, the measured thinking-token reduction and recommended sampling: see the main model card at bottlecapai/ThinkingCap-gemma-4-26B-A4B-it.

As in the base model, thinking is off unless the request turns it on (enable_thinking=True); the usage examples below turn it on.

Expected performance

ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf (MXFP4 / Q5_1 experts, Q8_0 attention, dense and embedding), served by llama.cpp, against the bf16 checkpoint of the main card, on nine of its benchmarks. AA-LCR, τ²-bench and LongBench v2 are not yet covered.

• bf16 is the main card's checkpoint on the same questions, every card seed averaged per question, so the MMLU-Pro and MMMLU subsets (uniform random samples of the full test sets) do not reproduce the card's full-set numbers exactly. Δ is GGUF minus bf16; the interval is a 95% bootstrap over questions. Thinking tokens count the reasoning trace only, over question × seed.
• The GGUF ran on llama.cpp bfd73a8 with the main card's sampling — thinking on, temperature 1.0, top_p 0.95, top_k 64 — and a 65,536-token cap.
• Note the unquantized BF16 GGUF on the same llama.cpp server thinks 6.1% longer than the bf16 checkpoint on AIME 2026 and 14.7% longer on GPQA-Diamond.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (LM Studio, Ollama, Jan, …). The quantized files below store the weights at 4.3 to 8.5 bits per weight instead of the checkpoint's 16.

Every file keeps the weights that run on every token at 8-bit Q8_0: the attention projections, the dense feed-forward block, and the token embedding, which is also the output head. The router stays in F32. The files differ in how they store the routed experts, which hold 90% of the 25.2B parameters while only 8 of 128 run per token; the last layer's experts are kept one step higher than the rest. MXFP4 is 4-bit floating point with a shared scale per block of 32 weights, the format family of NVFP4.

The Q6_K, Q4_K, Q5_1, IQ3_S and IQ4_NL tensors are quantized with an importance matrix (activation statistics from a chat-formatted calibration corpus with reasoning traces, which steer where the rounding error lands); it is included as ThinkingCap-gemma-4-26B-A4B-it-imatrix.gguf for anyone building other sizes.

Files

Usage (llama.cpp)

# serve an OpenAI-compatible endpoint, thinking on, with the base model's recommended sampling
llama-server -hf bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF:MXFP4_MOE \
  --jinja --chat-template-kwargs '{"enable_thinking":true}' --reasoning-format deepseek \
  --temp 1.0 --top-p 0.95 --top-k 64 -ub 2048 -b 2048

# or download the file and chat in the terminal
huggingface-cli download bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF \
  ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf --local-dir .
llama-cli -m ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' --temp 1.0 --top-p 0.95 --top-k 64

With --reasoning-format deepseek the server returns the thinking in reasoning_content and the answer in content. A client can also turn thinking on per request with "chat_template_kwargs": {"enable_thinking": true}. Keep the temperature at the recommended 1.0; greedy decoding can loop.

Vision (image input)

Image input needs the multimodal projector mmproj-ThinkingCap-gemma-4-26B-A4B-it-f16.gguf (in this repo) loaded alongside any of the text files. llama-server -hf … downloads and loads it automatically.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio detects it and enables image input.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF \
  ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf mmproj-ThinkingCap-gemma-4-26B-A4B-it-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-gemma-4-26B-A4B-it-MXFP4_MOE.gguf \
  --mmproj mmproj-ThinkingCap-gemma-4-26B-A4B-it-f16.gguf -ub 2048 -b 2048 \
  --image photo.jpg -p "Describe this image."

• llama-server: add --mmproj mmproj-ThinkingCap-gemma-4-26B-A4B-it-f16.gguf when serving a local file.

Keep -ub 2048 -b 2048 for image input. Gemma 4 attends bidirectionally across an image's tokens, so the whole image must fit in one micro-batch, and with llama.cpp's default of 512 a larger image aborts the server (GGML_ASSERT ... non-causal attention requires n_ubatch >= n_tokens, llama.cpp bfd73a8).

Where to find us

Need even more efficiency? Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Built for AI labs, inference providers and enterprises running models at scale. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

Released under the base model's own license, Apache 2.0 (see the Gemma 4 license page).