Gemma 4 12B IT GGUF
GGUF quantizations of Google's Gemma 4 12B IT model.
Optimized for local inference with:
- Ollama
- LM Studio
- Jan AI
- llama.cpp
- Open WebUI
- Text Generation WebUI
- KoboldCpp
Downloads
| Quant | Recommended Use |
|---|---|
| Q4_K_S | Smallest size |
| Q4_K_M | ⭐ Recommended |
| Q8_0 | Maximum quality |
| F16 | Further quantization |
Quick Start
Ollama
Create a file named Modelfile
FROM ./gemma-4-12B-it-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 32768
SYSTEM You are a helpful AI assistant.
Create model:
ollama create gemma4-12b -f Modelfile
Run:
ollama run gemma4-12b
LM Studio
- Download a GGUF file.
- Open LM Studio.
- Import the model.
- Load the model.
- Start chatting.
Recommended:
gemma-4-12B-it-Q4_K_M.gguf
Jan AI
- Download a GGUF file.
- Open Jan AI.
- Models → Import Model
- Select the GGUF file.
- Start chatting.
Recommended:
gemma-4-12B-it-Q4_K_M.gguf
Open WebUI
Using Ollama:
ollama create gemma4-12b -f Modelfile
Then select:
gemma4-12b
inside Open WebUI.
llama.cpp
Run:
./build/bin/llama-cli \
-m gemma-4-12B-it-Q4_K_M.gguf \
-cnv
Server:
./build/bin/llama-server \
-m gemma-4-12B-it-Q4_K_M.gguf
OpenAI-compatible endpoint:
http://localhost:8080/v1
Hardware Recommendations
| Quant | RAM |
|---|---|
| Q4_K_S | 10 GB+ |
| Q4_K_M | 12 GB+ |
| Q8_0 | 18 GB+ |
| F16 | 32 GB+ |
Which Version Should I Download?
Most Users
Q4_K_M
Best balance of:
- Quality
- Speed
- Memory usage
Highest Quality
Q8_0
Smallest Size
Q4_K_S
Further Quantization
F16
Credits
- Google for Gemma 4
- ggml-org for llama.cpp
- Ollama
- LM Studio
- Jan AI
- Open WebUI
Disclaimer
This repository contains GGUF conversions only.
All original model weights and intellectual property belong to Google.