Granite 4.2 3B + Uno drafter for the AMD XDNA 2 NPU

ibm-granite/granite-4.2-3b in OpenFlowLM's Q4NX format (q4_1), with its Uno adapter (uno.q4nx) and a prebuilt open kernel set. With all three, OpenFlowLM decodes greedy requests by Uno's two-pass cycle: one pass drafts 4 tokens, a second verifies them with the unmodified base. The output is identical to plain greedy decoding.

Needs an OpenFlowLM-Next build with PR #184 and PR #192 until they are merged. Without uno.q4nx, the same weights run as plain Granite.

Speed on the NPU

q4_1, 4 rows a pass, six prompts × 128 tokens, quiet machine (Ryzen AI HX 370).

All six are identical to plain decode. Geometric mean: 0.94x. A cycle costs about 3.2 decode steps and this adapter commits about 3 tokens a cycle at 4 rows, so Uno only wins on math and reasoning today. It should pay once a pass gets cheaper or 8-row passes run (3.53 tokens a cycle on the CPU reference).

Run

1. Download into a models directory:

   hf download Cyronius/Granite-4.2-3B-Uno-NPU2 --local-dir <root>/models/Granite-4.2-3B-Uno-NPU2
   

2. Add an entry under "models" in a copy of OpenFlowLM's model_list.json:

   "granite4.2": {"3b": {
     "name": "Granite-4.2-3B-Uno-NPU2", "url": "", "file_url": "", "ms_url": "",
     "modified_at": "2026-10-10T00:00:00Z", "oflm_min_version": "0.9.22",
     "files": ["config.json", "model.q4nx", "tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "uno.q4nx"],
     "default_context_length": 8192, "max_prefill_len": 4096,
     "details": {"family": "granite", "think": false, "think_toggleable": false,
                 "parameter_size": "3B", "quantization_level": "Q4_1"},
     "label": [], "footprint": 3.0}}
   

3. Point oflm at both and run:

   set OFLM_CONFIG_PATH=<your model_list.json>
   set OFLM_MODEL_PATH=<root>
   oflm run granite4.2:3b
   

   The kernels in open_kernels/ are found beside the model. Uno only runs greedy, and Granite samples by default, so type /set topk 1 and /set rep-pen 1 first. Over oflm serve, send "top_k": 1, "repetition_penalty": 1.

oflm add doesn't copy uno.q4nx or open_kernels/ yet, so use the steps above.

Files

Reproduce

q4nx-build -i granite-4.2-3b-bf16.gguf -s <granite-4.2-3b dir> -f granite -t language --quant Q4_1 \
  --uno-adapter <granite-4.2-3b-uno adapter dir> --uno-noise-high 100256 -o Granite-4.2-3B-Uno-NPU2
python open_kernels/export_qwen36_kernels.py --model-dir Granite-4.2-3B-Uno-NPU2

The GGUF is ibm-granite/granite-4.2-3b-GGUF's bf16 file. The procedure and checks are requirement OPEN-FAMILY-GRANITE-UNO in OpenFlowLM-Next's specs/open-engine/spec.md.

Credits

• Base model: IBM Granite 4.2 3B (Apache-2.0).
• Method: Uno by IFM (Apache-2.0, arXiv 2609.04010). The adapter was trained on OpenThoughts3 and Tülu 3 with AMD Developer Cloud compute.
• Runtime: OpenFlowLM-Next.