Qwen3-0.6B-int8-sym-ov

Qwen/Qwen3-0.6B (revision c1899de) as an OpenVINO IR with int8 symmetric per-channel weights, for the Intel NPU.

pip install "openvino-genai==2026.4.1.0" huggingface_hub
hf download mlboydaisuke/Qwen3-0.6B-int8-sym-ov --local-dir Qwen3-0.6B-int8-sym-ov

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("Qwen3-0.6B-int8-sym-ov", "NPU", GENERATE_HINT="BEST_PERF", NPUW_LLM_PREFILL_CHUNK_SIZE=64, CACHE_DIR="npu_cache")
print(pipe.generate("What is the capital of France? /no_think", max_new_tokens=32, do_sample=False))

Run it in the folder where you ran hf download; in a folder without Qwen3-0.6B-int8-sym-ov, LLMPipeline stops with Could not find a model in the directory. On the laptop below, it printed an empty <think> block, then The capital of France is **Paris**.

NPUW_LLM_PREFILL_CHUNK_SIZE=64 runs the prompt in chunks of 64 tokens instead of one pass padded to 1,024 tokens. The first token then comes sooner for a short prompt and later for a long one (numbers below). GENERATE_HINT="BEST_PERF" compiles the token-by-token stage for speed instead of for a fast compile (the default, FAST_COMPILE). The tokens after the first then come faster (numbers below). The first load, which compiles, takes longer: about 16 s here, against about 7 s with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone. CACHE_DIR keeps the compiled model, so later loads skip the compile. Without these settings the pipeline runs with OpenVINO GenAI's NPU defaults. do_sample=False decodes greedily, as in every number below; without it, generation follows this export's generation_config.json, which samples (temperature 0.6, top_k 20, top_p 0.95). The snippet decodes greedily only to match how those numbers were measured; for other uses, Qwen's Best Practices say "DO NOT use greedy decoding" in thinking mode and suggest Temperature=0.7, TopP=0.8, TopK=20, and MinP=0 in non-thinking mode.

Measured on one laptop's NPU

Intel Core Ultra 9 386H, Windows 11, Intel NPU driver 32.0.100.5540, OpenVINO GenAI 2026.4.1, MSI Center's Balanced mode, 2026-11-12. The times on this card, including the first load of about 16 s above, come from processes that turn Windows power throttling off (power_throttling in the speed run files); started over ssh with power throttling left to Windows, the snippet's first run took 50 s. The snippet was not timed from a terminal on the laptop itself.

• Answer agreement: the first line of this export's answer matched Qwen/Qwen3-0.6B's (fp32, transformers, CPU) on 18 of 20 short questions (thinking off). The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. The count is from the run of 2026-11-10 (records/cell_npu.json; its answers are in records/npu_timed_run1.json), on driver 32.0.100.4841. In the speed runs below, on driver 32.0.100.5540, every answer had the same tokens (new_ids) as in that run.
• Speed over those 20 questions and one more short prompt (21 prompts of 23–40 tokens, answers of 2–27 tokens): with the settings above, 35 ms to the first token, then 73.5 tokens/s; with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone, 35 ms, then 63.9 tokens/s; without either setting, 154 ms, then 64.9 tokens/s (in the same measuring session; each the median of 3 runs).
• Longer prompts take longer to the first token with NPUW_LLM_PREFILL_CHUNK_SIZE=64: 227 ms at 512 tokens and 493 ms at 1,000 tokens with the settings above (232 ms and 510 ms with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone; without either setting: 155 ms and 155 ms; median of 3 tries each, records/2026-11-12-best-perf/npu_long_prompt*.json).
• Long prompts give the same answers with NPUW_LLM_PREFILL_CHUNK_SIZE=64 as without it: on 107 of 108 prompts longer than one 64-token chunk (65–1,000 tokens, answers of up to 48 tokens), the greedy answer had the same tokens in two runs with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone and two without either setting. The other one is the same answer worded differently: to "What is the capital of France? Answer with one word." at the end of a 65-token prompt, the answer is "France's capital is Paris." with the setting and "Paris" without it (records/2026-11-12/npu_chunk64_answers.json). Adding GENERATE_HINT="BEST_PERF" changed no answer: in two runs with it, all 130 prompts of that check (the 108 above and 22 shorter ones) gave the same tokens as with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone (records/2026-11-12-best-perf/npu_best_perf_answers.json).
• In long prompts, the NPU got every fact right, as the CPU in f32 did: 160 prompts (8 short facts × 5 places × 4 lengths: 256, 512, 768 and 1,000 tokens) put one fact into a passage (as its first sentence, about a quarter, half or three quarters in, or near the end) and ask about it at the end (thinking off). With the settings above, this export answered all 160 correctly on the NPU (the same answer tokens as with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone and without either setting) and all 160 on the CPU in f32 (its properties are in the record); Qwen/Qwen3-0.6B in fp32 (transformers, CPU) answered all 160 too. The answers and the prompts: records/2026-11-12/npu_needle_answers.json.

Speed still being tuned (updates follow). Records: records/ (the speed runs above are in records/2026-11-12-best-perf/; tok_per_s in the run files is over whole answers, first token included).

Exported with optimum-cli export openvino --task text-generation-with-past --weight-format int8 --sym (optimum-intel 2.2.0, nncf 3.4.0, openvino 2026.4.1) from the files of Qwen/Qwen3-0.6B at revision c1899de, each checked against the Hub's sha256 (git blob sha1 for the small files).