SmolLM3-3B, OpenVINO INT4 channel-wise (NPU)

HuggingFaceTB/SmolLM3-3B at commit a07cc9a04f16550a088caea529712d1d335b0ac1, exported to OpenVINO IR with weights in symmetric, channel-wise INT4. Channel-wise (--group-size -1) is the layout the Intel NPU compiles quickly; a group-wise INT4 export of a similar model was still compiling after ten minutes.

Made for vinoWhisper, which uses it on the NPU to clean up dictation, and pins this repo by commit and per-file sha256.

How it was made

optimum-cli export openvino --model HuggingFaceTB/SmolLM3-3B \
  --task text-generation-with-past --weight-format int4 --sym --ratio 1.0 --group-size -1 out/

From a local download of the commit above. OpenVINO 2026.4.1, openvino-tokenizers 2026.4.1.0, optimum-intel 2.2.0, optimum 2.3.0, nncf 3.4.0, transformers 5.3.0, torch 2.14.1. Nothing else was changed.

Measured (2026-10-27, Intel Core Ultra NPU, 16 TOPS, OpenVINO GenAI 2026.4.1)

LLMPipeline(..., "NPU", MAX_PROMPT_LEN=3072, MIN_RESPONSE_LEN=512), greedy, /no_think:

One laptop, n=1 per figure. For summaries, set repetition_penalty=1.1: with greedy decoding and no penalty, one long summary looped.

Licence

Apache 2.0, the same as the base model.