GGUF quantizations of IBM Granite 4.1 8B Instruct, composed as a Granite Switch checkpoint with 13 embedded adapters (LoRA and aLoRA). The base weights are quantized with a Dutch importance matrix; all adapter weights stay at F16.
We are not aware of other published quantizations of a Granite Switch checkpoint with all 13 adapters, so treat this repository as experimental and read the Evaluation and Limitations sections before relying on it.
Status
- Released: Q6 (near-lossless), Q3 and Q2 (experimental).
- Not released: Q5 and Q4. In our tests the layer-map variant was worse than llama.cpp's own recipe at those two levels, and we do not yet understand why (see Evaluation).
- Size warning: every file carries a 5.54 GB F16 adapter stack, about 75% of which is zero padding. See the adapter stack before choosing a quant.
Sizes are decimal GB, as shown on Hugging Face. "Base part" is the file size minus the 5.54 GB adapter stack.
| File | Recipe | Size (GB) | Base part (GB) | Use |
|---|---|---|---|---|
granite-4.1-8b-13-adapters-Q6_K_XL.gguf |
layer map + Dutch imatrix | 12.77 | 7.23 | Recommended Q6 |
granite-4.1-8b-13-adapters-Q6_K.gguf |
llama.cpp recipe + Dutch imatrix | 12.42 | 6.88 | Reference for the layer map |
granite-4.1-8b-13-adapters-Q3_K_XL.gguf |
layer map + Dutch imatrix | 9.83 | 4.29 | Experimental, recommended Q3 |
granite-4.1-8b-13-adapters-Q3_K_M.gguf |
llama.cpp recipe + Dutch imatrix | 9.83 | 4.29 | Experimental reference |
granite-4.1-8b-13-adapters-Q2_K_XL.gguf |
layer map + Dutch imatrix | 8.99 | 3.45 | Experimental, recommended Q2 |
granite-4.1-8b-13-adapters-Q2_K.gguf |
llama.cpp recipe + Dutch imatrix | 9.02 | 3.48 | Experimental reference |
What the names mean. XL files take their per-tensor quantization types from Unsloth's UD-Qx_K_XL quant of Granite 4.1 8B. They are not Unsloth files: the weights are quantized here, with a Dutch imatrix, not Unsloth's calibration data. Q6_K, Q3_K_M and Q2_K are llama.cpp's own recipes (llama.cpp has no _M for Q6 and Q2). Both variants use the same Dutch imatrix; the only difference between the two is the per-tensor type map.
The 13 adapters add 5.54 GB (560 tensors: 14 per layer × 40 layers) to every file in this repository, at every quant level. For comparison, Unsloth's plain Granite 4.1 8B quants are 3.6 GB (Q2) to 7.9 GB (Q6). At Q2 the adapters are 62% of the file, at Q3 56%, at Q6 43%.
Why so large. Granite Switch stores each adapter projection as one stacked tensor of shape {n_embd, n_rank, n_slots}:
n_slots is the number of adapters plus one: slot 0 is all zeros and means "no adapter". With 13 adapters that is 14 slots.n_rank is the maximum rank of any adapter. Adapters with a lower rank are zero-padded up to it, and so are projections an adapter does not target.decisions adapter is rank 64, which raises the maximum rank, and therefore the size of every slot, to 64.A back-of-the-envelope calculation (14 slots × rank 64 × all seven projections × 40 layers, F16) gives 5.54 GB, which matches the measured size. The actual adapter weights are about 1.4 GB; roughly 4 GB are zeros. (These are estimates from IBM's published ranks and target modules; the per-adapter figures for this exact build have not been inspected.)
Speed. At run time only the selected slot is used, so decoding should cost about what a plain 8B costs, plus a small low-rank matrix multiplication per token. We have not benchmarked this.
VRAM. The zeros are stored and loaded like any other values; plan for memory equal to the file size. What the zeros take from you is context. A rough budget for a 16 GB card, assuming about 1 GB for compute buffers and the desktop, and a KV cache of about 160 KiB per token at F16 (40 layers × 8 KV heads × 128 dimensions; consistent with the adapter tensor sizes above):
| File | Left on 16 GB after weights and overhead | Context at F16 KV | Context at q8_0 KV | 12 GB card |
|---|---|---|---|---|
| Q6_K_XL (12.77 GB) | ≈ 2.2 GB | ≈ 13k tokens | ≈ 26k tokens | does not fit |
| Q3_K_XL (9.83 GB) | ≈ 5.2 GB | ≈ 32k tokens | ≈ 59k tokens | ≈ 7k tokens |
| Q2_K_XL (8.99 GB) | ≈ 6.0 GB | ≈ 37k tokens | ≈ 69k tokens | ≈ 12k tokens |
These are estimates, not measurements. 8 GB cards need partial offload for every file here.
Why this can still be a reasonable trade. The Switch pattern keeps follow-up calls (answerability, guardrails, decisions, citations) cheap by reusing the cached conversation, so a workflow can be built from short, separately logged steps rather than one very long context. Each step is an explicit control token plus, for the decision-style adapters, a single-token answer, which makes the trace easy to log and repeat. That may help traceability and reproducibility; this repository makes no compliance claims of any kind.
What could be done about the zeros (not done here, under consideration):
decisions (13 slots at rank 32) would need about 2.6 GB, saving roughly 3 GB per file.decisions on its own. The cost is that a decision call would not share the conversation's KV cache with the other adapters.The checkpoint is Granite 4.1 8B Instruct with 12 adapters from IBM's Granite Libraries (the same set as in IBM's Granite Switch 4.1 8B preview), plus the decisions adapter (a single-token decision engine: pick one option from a supplied set and get a probability for each). One adapter is active at a time, selected by a control token in the prompt.
| # | Adapter | Type | Control token | Library |
|---|---|---|---|---|
| 1 | citations | LoRA | <|citations|> |
granitelib-rag |
| 2 | query_rewrite | aLoRA | <|query_rewrite|> |
granitelib-rag |
| 3 | query_clarification | aLoRA | <|query_clarification|> |
granitelib-rag |
| 4 | hallucination_detection | LoRA | <|hallucination_detection|> |
granitelib-rag |
| 5 | answerability | aLoRA | <|answerability|> |
granitelib-rag |
| 6 | factuality-detection | aLoRA | <|factuality-detection|> |
granitelib-guardian |
| 7 | policy-guardrails | aLoRA | <|policy-guardrails|> |
granitelib-guardian |
| 8 | factuality-correction | aLoRA | <|factuality-correction|> |
granitelib-guardian |
| 9 | guardian-core | aLoRA | <|guardian-core|> |
granitelib-guardian |
| 10 | uncertainty | aLoRA | <|uncertainty|> |
granitelib-core |
| 11 | requirement-check | aLoRA | <|requirement-check|> |
granitelib-core |
| 12 | context-attribution | LoRA | <|context-attribution|> |
granitelib-core |
| 13 | decisions | aLoRA | see adapter_index.json |
granitelib-core |
Why the LoRA / aLoRA distinction matters in practice
llama-imatrix on the F16 model with a Dutch calibration text built from the first 2,500 texts of the Leesplank-vloeiend-nl-curriculum-cp2 dataset. The same imatrix is used for every file.XL files (layer map): the per-tensor quantization types of Unsloth's UD-Q6_K_XL, UD-Q3_K_XL and UD-Q2_K_XL GGUFs of Granite 4.1 8B were read with the gguf library and passed to llama-quantize --tensor-type-file. This maps 242 tensors. The Switch checkpoint fuses attention into 40 attn_qkv.weight tensors, which have no counterpart in Unsloth's separate attn_q/attn_k/attn_v; those follow llama.cpp's default rule for the chosen quant type.*.lora_a, *.lora_b) are kept at F16 in every file.quant4_1_v3_1.py.Method. Everything is measured against the F16 GGUF of the same checkpoint with llama-perplexity --kl-divergence, using one set of saved F16 reference logits for all files. Data: 70 clusters of Dutch text from the Leesplank corpus, 8 chunks of 512 tokens per cluster (about 143,000 scored token positions), using held-out texts only: 2,003 of the 17,500 cluster texts also occur in the imatrix calibration text and were excluded. Values are the mean over the 70 clusters; ± is the standard error over clusters. ΔPPL is the mean perplexity change relative to F16.
Noise floor. Running the F16 model twice with different GPU offload splits and comparing the two gives a mean KLD of 0.0012 and 98.2% identical top-1 tokens. Differences of that size are measurement noise.
| Quant | Recipe | Size (GB) | Mean KLD | Median KLD | Same top-1 token | ΔPPL vs F16 | Released |
|---|---|---|---|---|---|---|---|
| Q6_K_XL | layer map | 12.77 | 0.0069 ± 0.0002 | 0.0030 | 95.7% | −0.16% ± 0.05 | yes |
| Q6_K | llama.cpp | 12.42 | 0.0079 ± 0.0002 | 0.0036 | 95.3% | −0.05% ± 0.05 | yes |
| Q5_K_XL | layer map | 11.52 | 0.0181 ± 0.0005 | 0.0072 | 93.4% | +0.74% ± 0.07 | no |
| Q5_K_M | llama.cpp | 11.57 | 0.0141 ± 0.0004 | 0.0055 | 94.1% | +0.37% ± 0.07 | no |
| Q4_K_XL | layer map | 10.73 | 0.0434 ± 0.0013 | 0.0184 | 90.1% | +2.07% ± 0.14 | no |
| Q4_K_M | llama.cpp | 10.77 | 0.0395 ± 0.0013 | 0.0127 | 91.0% | +1.66% ± 0.13 | no |
| Q3_K_XL | layer map | 9.83 | 0.1125 ± 0.0037 | 0.0394 | 85.6% | +4.43% ± 0.28 | yes |
| Q3_K_M | llama.cpp | 9.83 | 0.1283 ± 0.0042 | 0.0427 | 85.0% | +5.45% ± 0.27 | yes |
| Q2_K_XL | layer map | 8.99 | 0.3357 ± 0.0104 | 0.1262 | 75.5% | +17.0% ± 0.7 | yes |
| Q2_K | llama.cpp | 9.02 | 0.3947 ± 0.0118 | 0.1533 | 74.2% | +22.6% ± 0.8 | yes |
What the numbers say
Only one adapter (answerability) was tested, with six cases (small English document, small Dutch document, and the 5,000-token document from the demo; one answerable and one unanswerable question each), using the layer-map files.
| Quant | Correct labels | Warm adapter call (after the base call filled the cache) |
|---|---|---|
| Q6_K_XL | 6 / 6 | 184 ms, 3 tokens processed, 5,043 from cache |
| Q3_K_XL | 6 / 6 | 90 ms, 3 tokens processed, 5,043 from cache |
| Q2_K_XL | 6 / 6 labels, but one reply ended with a stray <tool_call> token after the label |
91 ms, 3 tokens processed, 5,043 from cache |
The stray token at Q2 suggests that the adapter's stop behaviour is starting to degrade. Six cases is a smoke test, not a benchmark; the other twelve adapters have not been tested after quantization.
Use it for
Be aware
# llama.cpp (Vulkan / ROCm / CUDA / Metal builds with Granite Switch support)
llama-cli -m granite-4.1-8b-13-adapters-Q6_K_XL.gguf -ngl 99 \
--temp 0.7 \
-sp "Je bent een behulpzame assistent. Je antwoordt altijd in het Nederlands." \
-cnv
llama-server -m granite-4.1-8b-13-adapters-Q6_K_XL.gguf -ngl 99
Adapters are selected by a control token in the prompt. This demo sends the raw prompt to llama.cpp's /completion endpoint so the exact format is visible: the Granite Switch chat template, rendered by hand for one document and one question.
<|answerability|> control token instead of the normal assistant header. The aLoRA judges whether the question is answerable from the document. The prefix comes from the KV cache, so only 3 new tokens are processed.Start the server:
llama-server -m granite-4.1-8b-13-adapters-Q6_K_XL.gguf -ngl 99 -c 16384 --port 8080
Then paste this into a second terminal (needs bash, curl, jq). It needs no files: it builds its own test document with one fact buried in the middle.
# Self-contained demo document: filler + one real fact in the middle
DOC_JSON=$(awk 'BEGIN{for(i=1;i<=150;i++){
if(i==76) print "The Rotterdam office of the Delta Works Cooperative opened in 1987 and employs 240 people.";
else printf "Report %d describes maintenance cycle %d for pumping station %d, which moved %d cubic metres of water in week %d of the monitoring period.\n", i,i,i,i*37,i }}' | jq -Rs .)
Q="When did the Rotterdam office of the Delta Works Cooperative open, and how many people work there?"
PREFIX=$(cat <<EOF
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.
You are given a list of documents within <documents></documents> XML tags:
<documents>
{"doc_id": "1", "text": ${DOC_JSON}}
</documents>
Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>${Q}<|end_of_text|>
EOF
)
PREFIX+=$'\n' # $(...) strips the trailing newline the template requires
ask() { # $1 = prompt suffix, $2 = max tokens
jq -nc --arg p "${PREFIX}$1" --argjson n "$2" \
'{prompt:$p, n_predict:$n, temperature:0, cache_prompt:true}' |
curl -s http://localhost:8080/completion -H 'Content-Type: application/json' --data-binary @- |
jq '{answer:.content, processed:.timings.prompt_n, from_cache:.timings.cache_n,
total_ms:((.timings.prompt_ms + .timings.predicted_ms) | round)}'
}
# Call 1: base model answers the question (prefills the whole document)
ask '<|start_of_role|>assistant<|end_of_role|>' 64
# Call 2: answerability aLoRA on the same prefix (control token replaces the assistant role marker)
ask '<|answerability|>assistant<|end_of_role|>' 6
Or alternatively, on Windows:
# Self-contained demo document: filler + one real fact in the middle
$lines = foreach ($i in 1..150) {
if ($i -eq 76) {
"The Rotterdam office of the Delta Works Cooperative opened in 1987 and employs 240 people."
} else {
"Report $i describes maintenance cycle $i for pumping station $i, which moved $($i*37) cubic metres of water in week $i of the monitoring period."
}
}
$docText = [string]::Join("`n", $lines) + "`n"
$docJson = $docText | ConvertTo-Json -Compress
$Q = "When did the Rotterdam office of the Delta Works Cooperative open, and how many people work there?"
$prefix = @"
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.
You are given a list of documents within <documents></documents> XML tags:
<documents>
{"doc_id": "1", "text": $docJson}
</documents>
Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>$Q<|end_of_text|>
"@
$prefix += "`n"
# Helper function equivalent to the bash 'ask' function
function Invoke-LLMCompletion {
param([string]$Suffix, [int]$NPredict)
$body = @{
prompt = $prefix + $Suffix
n_predict = $NPredict
temperature = 0
cache_prompt = $true
} | ConvertTo-Json -Depth 10 -Compress
$res = Invoke-RestMethod -Uri "http://localhost:8080/completion" -Method Post -ContentType "application/json" -Body $body
[PSCustomObject]@{
answer = $res.content
processed = $res.timings.prompt_n
from_cache = $res.timings.cache_n
total_ms = [Math]::Round($res.timings.prompt_ms + $res.timings.predicted_ms)
}
}
# Call 1: base model answers the question (prefills the whole document)
Invoke-LLMCompletion -Suffix '<|start_of_role|>assistant<|end_of_role|>' -NPredict 64
# Call 2: answerability aLoRA on the same prefix (control token replaces the assistant role marker)
Invoke-LLMCompletion -Suffix '<|answerability|>assistant<|end_of_role|>' -NPredict 6
Example output (Q6_K_XL, ):
{
"answer": "The Rotterdam office of the Delta Works Cooperative opened in 1987 and employs 240 people.",
"processed": 5046,
"from_cache": 0,
"total_ms": 8431
}
{
"answer": "answerable",
"processed": 3,
"from_cache": 5043,
"total_ms": 139
}
The 5,043 cached tokens are the whole conversation so far: the adapter only switches on at its control token, so everything before it is the plain base model's cache. To see the other answer, set Q to a question the document cannot answer (for example "Who is the CEO?"), rebuild PREFIX and run both calls again; call 2 should return unanswerable.
Notes:
chat_template.jinja). For an aLoRA triggered by the assistant role, the control token takes the place of <|start_of_role|>. To check your own rendering, post the same messages to llama-server's /apply-template endpoint and compare.<requirements> for requirement-check, <guardian> for the guardian adapters), so check each adapter's template entry before calling it.cache_prompt on (the default in current llama.cpp).ibm-granite/granite-4.1-8b (Instruct), Apache-2.0granitelib-rag-r1.0, granitelib-core-r1.0, granitelib-guardian-r1.0) plus decisions from ibm-granite/granitelib-core-r1.0unsloth/granite-4.1-8b-GGUF, files granite-4.1-8b-UD-Q6_K_XL.gguf, granite-4.1-8b-UD-Q3_K_XL.gguf, granite-4.1-8b-UD-Q2_K_XL.ggufMichielBuisman/Leesplank-vloeiend-nl-curriculum-cp2quant4_1_v3_1.py (descendant of quant_one_model.py from the Granite 4.0 Dutch series)| Model | Quant | Status |
|---|---|---|
| Granite 4.1 8B Instruct + 13 Switch adapters | Q6_K / Q6_K_XL | released |
| Granite 4.1 8B Instruct + 13 Switch adapters | Q3, Q2 | released, experimental |
| Granite 4.1 8B Instruct + 13 Switch adapters | Q5, Q4 | withheld, under investigation |
Slim build without decisions |
— | under consideration |
| Granite 4.1 3B + Switch adapters | — | planned |
ibm-granite/granite-4.1-8b