Granite-4.0-H-Small — APEX GGUF
MoE-aware, mixed-precision APEX quantizations of
ibm-granite/granite-4.0-h-small
— IBM's hybrid Mamba-2 / Transformer MoE (granitemoehybrid): 40 layers
(36 Mamba-2 + 4 attention), 64-routed + shared-expert MoE, ~32B total / ~9B active,
Apache-2.0. To my knowledge the first APEX quant of a 32B Granite hybrid.
Results & which file to pick
Perplexity on wikitext-2-raw (test, 200×512-token windows), llama-perplexity.
All quants are imatrix-guided (diverse calibration).
| File | Size | BPW | PPL | Δ vs bf16 | Fits (VRAM) |
|---|---|---|---|---|---|
| bf16 (reference) | 61 GB | 16.0 | 5.752 | — | — |
| APEX-i-quality | 21 GB | 5.44 | 5.781 | +0.50% | 24 GB — best quality |
| APEX-i-compact | 16 GB | 4.13 | 5.998 | +4.28% | 24 GB |
| APEX-i-mini | 13 GB | 3.34 | 6.471 | +12.49% | 16 GB — smallest, quality trade |
- i-quality — recommended; near-lossless (+0.5%). Q6_K/Q5_K experts, Q8_0 shared, Q6_K attn/SSM.
- i-compact — Q3_K mid experts; a real but usable quality step (+4.3%) for tighter fits.
- i-mini — IQ2_S mid experts; smallest that stays coherent (16 GB cards), but a clear quality drop (+12.5%) — pick it only when you must fit and can tolerate the hit.
Note on minimum size: Granite's routed-expert dim is 640 (not 256-divisible), so i-quants can't apply to the
ffn_down_exps— they fall back to ~Q4_K. That floors the practical minimum around ~13 GB (i-mini) and, as a bonus, protects the down projections' quality. Going below that (nano/IQ1) buys little here.
Usage (llama.cpp)
llama-cli -m granite-4.0-h-small-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-small-APEX-i-compact.gguf -ngl 999 --host 0.0.0.0 --port 8080
Requires a llama.cpp build supporting the granitemoehybrid architecture.
Method
APEX = per-tensor-role, per-layer precision over stock llama-quantize --tensor-type-file.
Granite-H needed its Mamba-2 mixer tensors added to the map (the stock generator omits them):
- Mamba-2 (36 layers):
ssm_in,ssm_conv1d,ssm_outat mixer precision; 1-D state (ssm_a/d/dt/norm) left F32. - Attention (4 layers):
attn_q/k/v/output. - MoE (all 40): routed
ffn_*_expson a depth gradient; sharedffn_*_shexpat Q8_0/Q6_K.
Baseline: IBM's own f16 GGUF. imatrix from Bartowski calibration_datav3.
Config gen + patcher: configs/, patch_granite_config.py, REPRODUCE.md.
Null result: protecting the Mamba-2 recurrence doesn't help
A hand-roll pinning ssm_in/out/conv1d to Q8_0 (22 GB, PPL 5.779) was within noise of
i-quality — the SSM/recurrence tensors aren't precision-sensitive. Consistent with the
same finding on Kimi-Linear's KDA and Granite-4.0-H-Tiny.
It is shipped here as granite-4.0-h-small-APEX-handroll.gguf for transparency and
reproduction, but there's no reason to use it — it's slightly larger than i-quality
(22 GB vs 21 GB) for no measurable quality gain (PPL 5.779 vs 5.781). Use i-quality.
Attribution & licenses
See LICENSE (Apache-2.0) and NOTICE.
- Base: IBM (@ibm-granite) — granite-4.0-h-small (Apache-2.0)
- Engine: llama.cpp (@ggml-org) (MIT)
- APEX: Ettore Di Giacinto / LocalAI (@mudler) — localai-org/apex-quant (MIT)
- Calibration: Bartowski (@bartowski) — calibration_datav3
Unofficial community quantization; not affiliated with or endorsed by IBM.