Granite 4.2 8B · GGUF Quantizations
Ready-to-run GGUF quantizations of IBM Granite 4.2 8B for local inference with llama.cpp, Ollama, LM Studio, Open WebUI, and other GGUF-compatible runtimes.
This repository provides a complete range of quantization formats, from near-lossless BF16 and Q8 models to highly compressed IQ variants for memory-constrained devices.
About the Base Model
Granite 4.2 8B is IBM's dense reasoning language model designed for agentic workflows, coding, tool use, structured outputs, and enterprise-grade reasoning.
Key Properties
| Property | Value |
|---|---|
| Base Model | ibm-granite/granite-4.2-8b |
| Parameters | ~8B |
| Architecture | GraniteForCausalLM |
| Context Window | 128K tokens |
| Reasoning Mode | Switchable thinking / non-thinking |
| Tool Calling | Supported |
| License | Apache 2.0 |
| Format | GGUF |
Features
- 🧠 Native reasoning with optional
<think>mode - 🛠️ Tool calling and agent workflows
- 💻 Strong coding and software engineering capabilities
- 📊 Structured JSON generation
- ⚡ Compatible with llama.cpp and GGUF ecosystems
- 📱 Efficient local deployment on consumer hardware
- 🌍 Multilingual support
Recommended Quantizations
| Quant | Recommendation |
|---|---|
| Q6_K | Highest quality without full precision |
| Q5_K_M | Excellent balance of quality and VRAM |
| Q4_K_M | Default recommendation for most users |
| IQ4_XS | Compact deployment with good quality |
| Q3_K_M | Low-memory environments |
| Q2_K | Maximum compression |
Usage with llama.cpp
./llama-cli \
-m granite-4.2-8b-Q4_K_M.gguf \
-c 8192 \
-p "Explain transformer attention."
Prompt Format
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Explain transformer attention.
<|im_end|>
<|im_start|>assistant
Hardware Guidance
| Quant | Approximate RAM / VRAM |
|---|---|
| Q2_K | 4 GB |
| Q3_K_M | 5 GB |
| Q4_K_M | 6 GB |
| Q5_K_M | 7 GB |
| Q6_K | 9 GB |
| Q8_0 | 11 GB |
| BF16 | 18+ GB |
Actual memory consumption depends on context length, batch size, and runtime configuration.
Intended Use
This model is suitable for:
- Conversational AI
- Coding assistants
- Software engineering agents
- Retrieval-Augmented Generation (RAG)
- Local AI deployments
- Research and experimentation
- Tool-calling workflows
Limitations
- Quantization may slightly reduce model quality compared to the original BF16 checkpoint.
- Smaller quantizations prioritize memory efficiency over accuracy.
- Model outputs may contain inaccuracies or hallucinations.
- Performance depends heavily on the selected quantization level and hardware.
License
This repository distributes quantized versions of IBM Granite 4.2 8B.
The original model is licensed under Apache 2.0. Please refer to the upstream model card for complete licensing and usage information.
Acknowledgements
- IBM Granite Team for the original Granite 4.2 model.
- llama.cpp contributors for GGUF support and quantization tooling.
- Hugging Face for model hosting and distribution.
Citation
If you use these quantizations in research or production, please cite the original Granite 4.2 model from IBM.