Gemma 4 26B-A4B-it GGUF — Quantized by BatiAI

Optimized GGUF quantizations of google/gemma-4-26B-A4B-it for on-device AI on Mac. Built directly from official Google BF16 weights by BatiAI for BatiFlow — free, unlimited, on-device AI automation.

Gemma 4 26B-A4B is a Mixture-of-Experts model (26B total parameters, ~3.8B active per token) with native multilingual + vision capabilities. Optimized for chat, code, tool calling, and now image understanding (via opt-in mmproj — see Two modes below).

Quick Start

# 24 GB Mac — Best quality / speed (recommended)
ollama pull batiai/gemma4-26b:iq4

# 24 GB Mac — Smaller, imatrix-calibrated
ollama pull batiai/gemma4-26b:iq3

# 24 GB Mac — Standard 3-bit
ollama pull batiai/gemma4-26b:q3

# 32 GB+ Mac — Higher quality
ollama pull batiai/gemma4-26b:q4

# 36 GB+ Mac — Highest on-device quality
ollama pull batiai/gemma4-26b:q6

Available Quantizations

Note for 16 GB Macs: 26B MoE loads but runs at ~0.3 t/s due to swap. For 16 GB Macs use batiai/gemma4-e4b (5 GB, ~57 t/s) instead.

Two modes — text-only by default, multimodal opt-in

Upstream Gemma 4 26B-A4B-it is multimodal (text + image + video understanding — but no audio; for audio see the smaller E2B / E4B variants). In the GGUF ecosystem this is delivered as two files: a main model.gguf (text tower) and a separate mmproj.gguf (multi-modal projector — the vision tower). We ship both, separate, so you can pick:

This is the same pattern unsloth / bartowski / mradermacher use for multimodal models — text-only on Ollama, full multimodal via llama.cpp + mmproj.

Multimodal usage (llama.cpp)

Download the main GGUF + the mmproj file:

# Pick a main model (text tower)
wget https://huggingface.co/batiai/Gemma-4-26B-A4B-it-GGUF/resolve/main/google-gemma-4-26B-A4B-it-IQ4_XS.gguf

# Pick the mmproj (vision tower) — Q6_K is the sweet spot, BF16 if you want zero loss
wget https://huggingface.co/batiai/Gemma-4-26B-A4B-it-GGUF/resolve/main/mmproj-Q6_K.gguf

Server mode (OpenAI-compatible Vision API):

llama-server \
  -m google-gemma-4-26B-A4B-it-IQ4_XS.gguf \
  --mmproj mmproj-Q6_K.gguf \
  -c 32768 --host 127.0.0.1 --port 8080

# Then post images via the OpenAI Vision API shape
curl http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
      {"type": "text", "text": "What does this screenshot show?"}
    ]
  }]
}'

One-shot CLI:

llama-mtmd-cli \
  -m google-gemma-4-26B-A4B-it-IQ4_XS.gguf \
  --mmproj mmproj-Q6_K.gguf \
  --image ~/Desktop/photo.jpg \
  -p "describe this image"

mmproj quantizations available

(Q8_0 not available because some vision tensors have shapes incompatible with Q8_0's column alignment requirement — applies to every quantizer of this model. Q6_K's K-quant block layout handles them.)

Benchmarks

MacBook Pro M4 Pro 48 GB — consumer Mac (real-user measurement)

BatiAI IQ4 outperforms Ollama's official 26B on the same hardware — both in raw speed (58–63 vs 56 t/s) and memory efficiency (15.1 vs 19.3 GB VRAM).

MacBook Pro M4 Max 128 GB — developer Mac

Mac mini M4 16 GB — boundary case

26B MoE on 16 GB Macs hits swap (model + macOS overhead exceeds 16 GB). Recommendation: 24 GB+ Mac. For 16 GB devices, see the smaller Gemma 4 E series below.

RAM Requirements

Why no Q2? (benchmark evidence)

We tested Q2_K extensively. It produces broken, unusable output on 26B MoE — infinite repetition loops. At 2-bit precision the MoE expert routing weights lose too much information. Q3 (3-bit) is the minimum viable quantization for this model.

Why BatiAI?

About BatiFlow

flow.bati.ai

BatiFlow is a macOS-native AI desktop automation app — just 5 MB, built with Swift.

• Free & Unlimited — On-device AI via Ollama, no API costs
• 100 % Private — All data stays on your Mac
• Ultra Lightweight — Native macOS app, only 5 MB
• 57 built-in tools — calendar, notes, reminders, files, email, browser, messaging, and more

Related models in the BatiAI lineup

Technical Details

• Original Model: google/gemma-4-26B-A4B-it
• Architecture: Gemma 4 Mixture-of-Experts (26 B total, ~3.8 B active per token)
• Modalities: Text (primary) + Image / Video via opt-in mmproj. Audio not supported (use E2B / E4B for audio)
• Context Window: 128 K tokens
• License: Apache 2.0
• Quantized with: llama.cpp
• Quantized by: BatiAI
• GGUF metadata: general.author: BatiAI, general.url: https://flow.bati.ai

How we quantize

Google official weights (BF16, ~50 GB)
  ↓ llama.cpp convert_hf_to_gguf.py
BF16 GGUF (~50 GB)
  ↓ llama-imatrix (calibration data)
Importance Matrix (imatrix.dat)
  ↓ llama-quantize  (Q3_K_M, Q4_K_M, Q6_K)
  ↓ llama-quantize --imatrix  (IQ3_M, IQ4_XS)
Quantized GGUF
  ↓ convert_hf_to_gguf.py --mmproj  (vision tower)
mmproj BF16 → Q6_K
  ↓ scripts/sign-batiai.py  (general.author = BatiAI)
Signed GGUF
  ↓ ollama push (text-only)  +  hf upload (text + mmproj)
Published

No third-party intermediaries. Direct from source, verified on real hardware.

License

This model is quantized from google/gemma-4-26B-A4B-it under the Apache 2.0 license. Commercial use permitted per upstream terms.

BatiAI's quantization pipeline is provided under MIT.

Benchmarks