Gemma 4 E2B Instruct — W4A16 Quantized AutoRound

This repository hosts W4A16 INT4-quantized versions of google/gemma-4-E2B-it, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available:

Note on MTP / Speculative Decoding:
If you want to use the official speculative decoding assistant model (google/gemma-4-E2B-it-assistant) for MTP support, it is recommended to use the vllm/vllm-openai:gemma4-0505-cu129 Docker image, which includes newer Gemma 4 support and decoding patches.

Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized google/gemma-4-E2B-it model.

Note: Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.

---

Quantization Details

---

Model Architecture

Gemma 4 E2B is a multimodal MoE model (Gemma4ForConditionalGeneration) with:

---

Usage

vLLM Inference

The recommended way to serve this model is via the official vllm/vllm-openai:gemma4 Docker image, which ships vLLM v0.19.1 with the latest Transformers patches required for Gemma 4.

Serve with Docker (recommended)

docker run --gpus all --rm -p 8000:8000 \
  vllm/vllm-openai:gemma4 \
  vllm serve Vishva007/gemma-4-E2B-it-W4A16-AutoRound \
    --served-model-name Gemma-4-E2B-it \
    --quantization autoround \
    --kv-cache-dtype auto \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --dtype bfloat16 \
    --max-model-len 18432 \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --port 8000 \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --mm-processor-kwargs '{"max_soft_tokens": 560}'

Direct vllm serve (vLLM ≥ 0.19.0)

vllm serve Vishva007/gemma-4-E2B-it-W4A16-AutoRound \
  --served-model-name Gemma-4-E2B-it \
  --quantization autoround \
  --kv-cache-dtype auto \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --dtype bfloat16 \
  --max-model-len 18432 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --port 8000 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --mm-processor-kwargs '{"max_soft_tokens": 560}'

max_soft_tokens — Image Token Budget

The max_soft_tokens parameter controls how many visual tokens are allocated per image. Higher values give richer image representations at the cost of context length and throughput.

Pass it via --mm-processor-kwargs '{"max_soft_tokens": <value>}'.

OpenAI-compatible API call

Once the server is running, query it like any OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see."},
                {"type": "image_url", "image_url": {"url": "https://..."}},
            ],
        }
    ],
    max_tokens=512,
)
print(response.choices.message.content)

The quantized model was then exported in both AutoRound and GPTQ formats and pushed to Hugging Face Hub.

---

Limitations & Notes

• RTN mode (iters=0) is used instead of full AutoRound optimization due to Gemma 4's architecture constraints.
• Some layers with shapes not divisible by 32 are skipped during quantization (minor precision impact).
• Multimodal (vision/audio) capabilities are fully preserved as those towers are not quantized.

---

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.15

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

---

Acknowledgements

Special thanks to OLAF-OSS and the contributors of the gemma4-vllm project for their excellent work in enabling Gemma 4 support with vLLM. Their QUANTIZE.md guide was extremely helpful in understanding the correct quantization approach for gemma-4-E2B-it.

At the time of writing, the original repository appears to be unavailable or removed. To ensure reproducibility, I have documented the full quantization workflow used for this model in my own repository.

The full quantization process used to produce these models is documented here: 📓 auto_round_Gemma4-E2B.ipynb

License

This quantized model is derived from google/gemma-4-E2B-it and is subject to the Gemma Terms of Use.

---

Citation

If you use this quantized model, please also cite the original Gemma 4 work:

@misc{gemma4_2026,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-E2B-it}
}

---

Quantized by Vishva007