Quantized variants of IBM Granite 4.0 Hybrid Micro, using a Dutch importance matrix derived from the Leesplank corpus.
For deployment, use the _NL variants. The _plain variants exist only to validate the contribution of the Unsloth layer map — see below.
For full methodology, see the dense micro model card and the quantization pipeline (quant_one_model.py).
| File | Size (approx) | Use |
|---|---|---|
granite-4.0-h-micro-q4_K_NL.gguf |
~4.5 GB | ✅ Recommended deployment |
granite-4.0-h-micro-q4_K_plain.gguf |
~4.4 GB | 🔬 Validation only |
Each _NL quantization combines two things:
UD-Q4_K_XL GGUF for this architecture, which promotes sensitive tensors (attention values, norms, selected FFN layers) to higher precision above the Q4_K_M base.The _plain variants use the Dutch imatrix with standard Q4_K_M quantization and no layer promotion. They exist to isolate how much of the quality gain comes from the layer map vs the imatrix alone.
This model is Granite 4.0 Hybrid Micro — a 3B-parameter SSM/attention hybrid using interleaved Mamba-2 recurrent layers alongside selective attention layers. This is architecturally distinct from the 3B dense micro, which uses standard transformer attention throughout.
From a quantization perspective this matters because SSM recurrent layers have different weight sensitivity profiles than attention layers. Mamba-2 state matrices accumulate small rounding errors across token positions in ways that transformer softmax can partially self-correct for — which might suggest hybrid models would be harder to quantize well. The data below shows the opposite: the hybrid micro actually degrades less relative to its FP16 baseline than the dense micro at Q4, suggesting that the recurrent layers are less weight-sensitive at this scale than expected, and that the Dutch imatrix calibration is effective across both layer types.
Evaluated on held-out Dutch texts from the same stratified corpus used for calibration. FP16 baseline mean PPL: 10.892, median: 11.660.
| Model | Mean PPL | Median PPL | Δ mean vs FP16 | Mean % Δ |
|---|---|---|---|---|
| FP16 baseline | 10.892 | 11.660 | — | — |
| Q4_K_NL (this model) | 11.037 | 11.870 | +0.145 | +1.33% |
| Q4_K_plain | 11.060 | 11.876 | +0.168 | +1.55% |
q_nl beats q_plain in 54 out of 70 clusters (77%). The layer map reduces the mean PPL delta by 14% relative to calibration-alone.
The dense 3B micro achieves a mean % delta of +1.50% with Q4_K_NL. The hybrid 3B micro achieves +1.33% — degrading less in relative terms despite the more complex hybrid architecture. The layer map contribution is smaller (14% vs 23% for dense), suggesting the Dutch imatrix alone is doing more relative work on the hybrid, possibly because the SSM layers are already less sensitive at this bit depth.
Note that the FP16 baselines are not directly comparable: the hybrid's higher absolute PPL (10.89 vs 8.94 for dense) reflects architectural and capacity differences, not lower Dutch language quality. The relative degradation percentages are the right metric to compare.
KLD measures how closely the full output probability distribution matches FP16 at each token position — more sensitive than perplexity, which only captures the most likely token.
| Model | Avg mean KLD | Avg median KLD | Avg flip rate | Avg p99 KLD |
|---|---|---|---|---|
| Q4_K_NL (this model) | 0.0489 | 0.0363 | 15.0% | 0.192 |
| Q4_K_plain | 0.0566 | 0.0417 | 16.2% | 0.231 |
q_nl reduces the average mean KLD by 11.8% over q_plain. The flip rate (fraction of positions where the top-1 token changes vs FP16) drops from 16.2% to 15.0%.
| Model | Worst chunk | Mean KLD | Flip rate | p99 KLD |
|---|---|---|---|---|
| Q4_K_NL | 025 | 0.104 | 0.250 | 0.457 |
| Q4_K_plain | 050 | 0.155 | 0.320 | 1.148 |
Chunk 050 is a notable anomaly for q_plain: despite a very low FP16 perplexity (2.99 — highly structured/predictable Dutch text), q_plain produces a p99 KLD of 1.148 and a 32% flip rate — a distribution collapse in the probability tails that perplexity alone would not reveal. q_nl substantially rescues this chunk (mean KLD 0.082, p99 0.383, flip rate 24%), demonstrating the value of Dutch imatrix calibration on structured Dutch text specifically. This is the exact failure mode imatrix was designed to prevent: quantization artifacts that corrupt the tail of the distribution on predictable, high-confidence sequences.
# llama.cpp
llama-cli -m granite-4.0-h-micro-q4_K_NL.gguf -ngl 99 -p "Vertel me over ..."
# LM Studio: load directly, select Granite instruct chat template
| Setting | Value |
|---|---|
| Context length | 128K (as per base model) |
| Chat template | Granite instruct |
| GPU offload | All layers (-ngl 99) |
This model is part of a series of Dutch-calibrated Granite GGUF quantizations:
| Model | Architecture | Size | Card |
|---|---|---|---|
| granite-4.0-tiny-dutch-calibrated-gguf | hybrid 7B | ~4.2 GB (Q4) | link |
| granite-4.0-micro-dutch-calibrated-gguf | Dense 3B | ~2.0 GB (Q4) | link |
| granite-4.0-h-micro-dutch-calibrated-gguf | Hybrid 3B | ~2.0 GB (Q4) | this card |
| granite-4.0-h-1b-dutch-calibrated-gguf | Hybrid 1B | ~0.8 GB (Q4) | link |
| granite-4.0-h-350m-dutch-calibrated-gguf | Hybrid 350M | ~0.25 GB (Q4) | link |
The dense micro card contains the full methodology description, evaluation design, and discussion of the Unsloth layer map extraction approach.
quant_one_model.py)ibm-granite/granite-4.0-h-micro