granite-3.1-1b-a400m-instruct — APEX quants

Measured per-tensor bit allocation (APEX). Curated: only sizes that clear the quality bar (PPL ≤ 1.5× fp) are published — we hold back quants that would misrepresent the line, rather than ship everything.

Held (below quality bar, not shipped): subcompact.

* Tool-call scores require granite-3.1-tools.jinja — see the warning above.

⚠️ Tool calling needs the Jinja file in this repo

granite-3.1 ships with a chat template that cannot emit tool calls, and it fails silently. Use granite-3.1-tools.jinja from this repo:

llama-server --jinja --chat-template-file granite-3.1-tools.jinja \
  --model granite-3.1-1b-a400m-instruct-APEX-quality.gguf \
  --ctx-size 8192 -fa on --temp 0.6 --top-p 0.9 --repeat-penalty 1.05

Without that flag there is no error and no warning. The model produces a perfectly correct tool call — it just arrives as raw text in content instead of the structured field, so an OpenAI-compatible client sees a chat message full of angle brackets, never sees a tool call, and the agent loop quietly stops. Measured on stock llama.cpp b2276-22397c31a, same model, same request, only the template differing:

So the model is not the problem and the quant is not the problem — the shipped template simply never declares the tool-call format for llama.cpp to parse. The 5/5 tool-call scores in the table below were measured with the repo's Jinja file, and are not reproducible without it.

Full round-trip you can paste and run: TOOL_CALLING.md.

See also

• Qualitative head-to-head
• Tool-calling demo (full transcript)

Attribution & licenses

• Base: ibm-granite/granite-3.1-1b-a400m-instruct (apache-2.0)
• Engine: llama.cpp (MIT)
• APEX: localai-org/apex-quant (@mudler) (MIT)
• Calibration: Salesforce/wikitext

Unofficial community quantization; not affiliated with or endorsed by the base model's publisher.