Sign up

bombman/granite-4.1-8b-GGUF

🤗 Hugging Face sourceapache-2.0-GGUF✓ HF checksums availableupdated today
No torrent yet →

Granite 4.1 8B GGUF (llama.cpp / LM Studio / Ollama / Jan.ai)

GGUF conversion of IBM Granite 4.1 8B for efficient local inference across multiple runtimes.

Supported runtimes:

  • llama.cpp
  • LM Studio
  • Ollama (via Modelfile)
  • Jan.ai

Model Overview

  • Base model: ibm-granite/granite-4.1-8b
  • Architecture: decoder-only transformer
  • Context length: 131072 tokens
  • License: Apache 2.0

Granite 4.1 8B is optimized for practical workloads such as code generation, structured output, tool usage, and retrieval-based systems.

Available Files

  • granite-4.1-8b.Q4_K_M.gguf (~5 GB) Recommended
  • granite-4.1-8b.Q4_K_S.gguf (~4.5 GB) Faster, lower quality
  • granite-4.1-8b.Q8_0.gguf (~8 GB) Higher quality
  • granite-4.1-8b-f16.gguf (~16 GB) Full precision

llama.cpp

Server: ./llama-server
-m granite-4.1-8b.Q4_K_M.gguf
-c 131072
--threads 8
--n-gpu-layers 35

CLI: ./main
-m granite-4.1-8b.Q4_K_M.gguf
-n 512
-c 131072
--temp 0.2
--top-p 0.9

LM Studio

  1. Open LM Studio
  2. Go to "Local Models"
  3. Import the .gguf file
  4. Select the model and start a chat or server

Recommended settings:

  • Context length: 131072
  • Temperature: 0.2
  • Top-p: 0.9

Ollama

Create a Modelfile:

FROM ./granite-4.1-8b.Q4_K_M.gguf

PARAMETER temperature 0.2 PARAMETER top_p 0.9 PARAMETER num_ctx 131072

Build and run:

ollama create granite-4.1-8b -f Modelfile ollama run granite-4.1-8b

Jan.ai

  1. Open Jan.ai
  2. Go to "Models"
  3. Click "Add Model" -> "Import GGUF"
  4. Select the .gguf file

Recommended settings:

  • Context length: 131072
  • Temperature: 0.2
  • Top-p: 0.9

Prompting Notes

Use explicit and structured instructions for best results.

Example:

You are a precise software engineer.

Task: Generate a REST API in Go.

Requirements:

  • Use standard library only
  • Return JSON responses
  • Include error handling

For structured output:

Return output as valid JSON only.

Performance Characteristics

  • Strong at code generation
  • Reliable structured / JSON output
  • Efficient for long-context tasks
  • Suitable for tool-calling workflows

Limitations:

  • Not optimized for deep reasoning
  • Average creative writing performance

Recommended Use Cases

  • Local coding assistant
  • AI agents
  • RAG systems
  • Backend LLM services

Conversion Notes

Unofficial GGUF conversion from the original Hugging Face model using llama.cpp tooling.

License

Apache 2.0 (same as upstream)

Acknowledgements

  • IBM Granite team
  • llama.cpp contributors