gemma-4-12b-it-GGUF
Quantized GGUF versions of google/gemma-4-12b-it.
Works with llama.cpp, Ollama, LM Studio, and any GGUF-compatible runtime.
Benchmarked on Kaggle T4 GPU · June 10, 2026 · Built with quant-kit
⚖️ Quality — Perplexity on WikiText-2
Lower = closer to original FP16 quality.
| Quant | Perplexity (WikiText-2) ↓ |
|---|---|
Q4_K_M |
2651.1317 |
🧠 Downstream Benchmarks — Q4_K_M on T4 GPU
Evaluated using lm-evaluation-harness
| Benchmark | Score | What it measures |
|---|---|---|
| IFEval | 18.8% |
Instruction following accuracy |
⚡ Speed — Q4_K_M on Kaggle T4 GPU
| Context | Token Generation | Prompt Processing |
|---|---|---|
| 128 tokens | 25.57 tok/s | 2530.38 tok/s |
| 512 tokens | 23.06 tok/s | 6207.49 tok/s |
| 2048 tokens | 22.48 tok/s | 14000.27 tok/s |
🚀 How to Use
llama.cpp
./llama-cli -m gemma-4-12b-it-Q4_K_M.gguf -p "Your prompt" -n 512
Python
from llama_cpp import Llama
llm = Llama(model_path="./gemma-4-12b-it-Q4_K_M.gguf", n_gpu_layers=-1)
print(llm("Tell me about AI", max_tokens=256)["choices"][0]["text"])
Quantized with quant-kit