Granite-4.0-H-Tiny — APEX GGUF

MoE-aware, mixed-precision APEX quantization of ibm-granite/granite-4.0-h-tiny — IBM's hybrid Mamba-2 / Transformer MoE (granitemoehybrid): 40 layers = 36 Mamba-2 + 4 attention, a 64-routed + shared-expert MoE FFN on every layer, ~7B total / ~1B active, Apache-2.0.

To my knowledge this is the first APEX quant of a Granite hybrid Mamba-2/MoE. APEX assigns precision per tensor role and per layer; here that meant teaching the recipe about the Mamba-2 mixer tensors the stock generator doesn't know (see Method).

Quality gate (retrofit, added after initial publish): real-generation coherence check passed on both shipped quants, plus a 3-turn chained tool-calling test (a distractor tool offered but correctly never called) — 3/3 on both. PPL-ratio vs bf16 also re-verified independently. See GATE.md and TOOL_CALLING.md for the full results and a reproducible transcript.

Results

Perplexity on wikitext-2-raw (test, 200×512-token windows), llama-perplexity.

The torch-imatrix row is the same APEX i-quality recipe, quantized with an importance matrix computed independently in PyTorch (see The torch-imatrix variant) instead of llama-imatrix. Its PPL (8.864) is inside the ±0.11 error bar of both bf16 and the standard i-quality — i.e. the two imatrices are interchangeable in quality.

Within 0.38% of full-precision perplexity at ~3× smaller, and it runs comfortably on modest hardware (~14 GB bf16 → 4.4 GB). Coherent, ~110 tok/s on a single GPU. Built with a diverse imatrix (Bartowski calibration_datav3), full expert coverage.

Usage (llama.cpp)

llama-cli   -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 --host 0.0.0.0 --port 8080

Requires a llama.cpp build supporting the granitemoehybrid (a.k.a. granitehybrid) architecture.

Method

APEX is a bit-allocation recipe over stock llama-quantize --tensor-type-file. Granite-H needed the Mamba-2 mixer tensors added to the map, which the stock APEX generator omits:

• Mamba-2 (36 layers): ssm_in, ssm_conv1d, ssm_out at mixer precision (Q6_K); 1-D state (ssm_a, ssm_d, ssm_dt, ssm_norm) left F32.
• Attention (4 layers): standard attn_q/k/v/output (Q6_K).
• MoE (all 40 layers): routed ffn_*_exps on a layer-depth precision gradient (Q6_K/Q5_K/IQ4_XS), shared experts ffn_*_shexp at Q8_0 (always active → protect), router left high. Routed intermediate dim 512 is 256-divisible → no IQ4_NL workaround.

Config generation + patcher: configs/, patch_granite_config.py, REPRODUCE.md. Baseline: IBM's own bf16 GGUF.

Which file to pick, and a null result

Use i-quality — smaller and better. The hand-roll tier (also shipped) pins the Mamba-2 recurrence tensors (ssm_in/out/conv1d) to Q8_0 to test whether protecting the linear-recurrence state helps: it doesn't — it's larger and slightly worse (4.5 GB, PPL 8.913 vs 8.901). Same null result seen on Kimi-Linear's KDA, so across two hybrid architectures the SSM/recurrence tensors just aren't precision-sensitive. The hand-roll is included to document the experiment; i-quality is the recommended tier.

The torch-imatrix variant

granite-4.0-h-tiny-APEX-i-quality-torch.gguf (+ granite-4.0-h-tiny-torch.imatrix) is a companion build that answers one question: can the calibration imatrix be produced without llama.cpp, and does that change the result?

What it is. An importance matrix (imatrix) records, per weight tensor, the per-input-channel sum of squared activations over calibration text — it tells llama-quantize where to spend bits. Normally you get it from llama-imatrix, which needs the whole model resident in RAM+VRAM. Here it was instead computed directly from the Hugging Face model in PyTorch, by registering forward hooks on every matmul (attention, Mamba-2 in/out projections, router, shared experts, and per-expert routed FFNs) and accumulating Σx² as calibration text streams through. The generator is band-serialized: it loads a few decoder layers at a time, runs all calibration chunks through them, caches activations, frees them, and moves on — so peak VRAM is a few layers, not the whole model (this run peaked at ~6.4 GB on a 16 GB card, vs the ~14 GB a full-resident forward needs). That decouples imatrix generation from llama.cpp's memory model: you can calibrate a model far larger than your GPU by streaming it in bands.

Everything else is identical. Same bf16 base, same APEX --tensor-type-file recipe, same calibration_datav3, same 126×512 calibration windows. The only difference from the standard i-quality file is which tool produced the imatrix. The two quants therefore differ only in the tensors whose quantization consumes an imatrix (the K-/I-quant expert, attention and Mamba weights); the Q8_0 shared experts and F32 tensors are bit-identical.

Validation. The PyTorch imatrix matches llama-imatrix's output tensor-for- tensor (368/368 names) with median per-tensor correlation 0.995. The handful of lower-correlation tensors are the post-nonlinearity inputs (ffn_down_shexp, attn_output) where transformers' naive Mamba scan and SiLU-gated intermediates differ numerically from llama.cpp's kernels — a known wrinkle that does not affect quality, since the quantizer only needs relative per-channel importance. The proof is in the table above: the torch-imatrix quant scores PPL 8.864, inside the ±0.11 error bar of both bf16 and the standard i-quality. The two imatrices are interchangeable.

Pick whichever you like — they are equivalent in quality. The standard i-quality is the reference; the torch build is provided for anyone who wants a llama.cpp-free, VRAM-bounded path to the same result (e.g. calibrating very large MoEs on modest hardware).

Attribution & licenses

See LICENSE (Apache-2.0) and NOTICE.

• Base: IBM (@ibm-granite) — granite-4.0-h-tiny (Apache-2.0)
• Engine: llama.cpp (@ggml-org) (MIT)
• APEX: Ettore Di Giacinto / LocalAI (@mudler) — localai-org/apex-quant (MIT)
• Calibration: Bartowski (@bartowski) — calibration_datav3

Unofficial community quantization; not affiliated with or endorsed by IBM.