Qwen3-0.6B-int8-sym-ov

Qwen/Qwen3-0.6B (revision c1899de) as an OpenVINO IR with int8 symmetric per-channel weights, for the Intel NPU.

pip install "openvino-genai==2026.4.1.0" huggingface_hub
hf download mlboydaisuke/Qwen3-0.6B-int8-sym-ov --local-dir Qwen3-0.6B-int8-sym-ov

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("Qwen3-0.6B-int8-sym-ov", "NPU", GENERATE_HINT="BEST_PERF", NPUW_LLM_PREFILL_CHUNK_SIZE=64, CACHE_DIR="npu_cache")
print(pipe.generate("What is the capital of France? /no_think", max_new_tokens=32, do_sample=False))

Run it in the folder where you ran hf download; in a folder without Qwen3-0.6B-int8-sym-ov, LLMPipeline stops with Could not find a model in the directory. On the laptop below, it printed an empty <think> block, then The capital of France is **Paris**.

NPUW_LLM_PREFILL_CHUNK_SIZE=64 runs the prompt in chunks of 64 tokens instead of one pass padded to 1,024 tokens. The first token then comes sooner for a short prompt and later for a long one (numbers below). GENERATE_HINT="BEST_PERF" compiles the token-by-token stage for speed instead of for a fast compile (the default, FAST_COMPILE): the first load, which compiles, took about 16 s here, against about 7 s with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone. CACHE_DIR keeps the compiled model, so later loads skip the compile. Without these settings the pipeline runs with OpenVINO GenAI's NPU defaults. do_sample=False decodes greedily, as in every number below; without it, generation follows this export's generation_config.json, which samples (temperature 0.6, top_k 20, top_p 0.95).

Measured on one laptop's NPU

Intel Core Ultra 9 386H, Windows 11, Intel NPU driver 32.0.100.5540, OpenVINO GenAI 2026.4.1, MSI Center's Balanced mode, 2026-11-12:

Answer agreement: the first line of this export's answer matched Qwen/Qwen3-0.6B's (fp32, transformers, CPU) on 18 of 20 short questions (thinking off). The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. The count is from the run of 2026-11-10 (records/cell_npu.json; its answers are in records/npu_timed_run1.json), on driver 32.0.100.4841. In the speed runs below, on driver 32.0.100.5540, every answer had the same tokens (new_ids) as in that run.
Speed over those 20 questions and one more short prompt (21 prompts of 23–40 tokens, answers of 2–27 tokens): with the settings above, 35 ms to the first token, then 73.5 tokens/s; with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone, 35 ms, then 63.9 tokens/s; without either setting, 154 ms, then 64.9 tokens/s (in the same measuring session; each the median of 3 runs).
Longer prompts take longer to the first token with NPUW_LLM_PREFILL_CHUNK_SIZE=64: 227 ms at 512 tokens and 493 ms at 1,000 tokens with the settings above (232 ms and 510 ms with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone; without either setting: 155 ms and 155 ms; median of 3 tries each, records/2026-11-12-best-perf/npu_long_prompt*.json).
Long prompts give the same answers with NPUW_LLM_PREFILL_CHUNK_SIZE=64 as without it: on 107 of 108 prompts longer than one 64-token chunk (65–1,000 tokens, answers of up to 48 tokens), the greedy answer had the same tokens in two runs with the setting and two without. The other one is the same answer worded differently: to "What is the capital of France? Answer with one word." at the end of a 65-token prompt, the answer is "France's capital is Paris." with the setting and "Paris" without it (records/2026-11-12/npu_chunk64_answers.json). Adding GENERATE_HINT="BEST_PERF" changed no answer: in two runs with it, all 130 prompts of that check (the 108 above and 22 shorter ones) gave the same tokens as with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone (records/2026-11-12-best-perf/npu_best_perf_answers.json).

Speed still being tuned (updates follow). Records: records/ (the speed runs above are in records/2026-11-12-best-perf/; tok_per_s in the run files is over whole answers, first token included).

Exported with optimum-cli export openvino --task text-generation-with-past --weight-format int8 --sym (optimum-intel 2.2.0, nncf 3.4.0, openvino 2026.4.1) from the files of Qwen/Qwen3-0.6B at revision c1899de, each checked against the Hub's sha256 (git blob sha1 for the small files).

Loading…

Qwen3-0.6B-int8-sym-ov

Qwen/Qwen3-0.6B (revision c1899de) as an OpenVINO IR with int8 symmetric per-channel weights, for the Intel NPU.

pip install "openvino-genai==2026.4.1.0" huggingface_hub hf download mlboydaisuke/Qwen3-0.6B-int8-sym-ov --local-dir Qwen3-0.6B-int8-sym-ov

import openvino_genai as ov_genai pipe = ov_genai.LLMPipeline("Qwen3-0.6B-int8-sym-ov", "NPU", GENERATE_HINT="BEST_PERF", NPUW_LLM_PREFILL_CHUNK_SIZE=64, CACHE_DIR="npu_cache") print(pipe.generate("What is the capital of France? /no_think", max_new_tokens=32, do_sample=False))

Measured on one laptop's NPU

Intel Core Ultra 9 386H, Windows 11, Intel NPU driver 32.0.100.5540, OpenVINO GenAI 2026.4.1, MSI Center's Balanced mode, 2026-11-12:

Answer agreement: the first line of this export's answer matched Qwen/Qwen3-0.6B's (fp32, transformers, CPU) on 18 of 20 short questions (thinking off). The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. The count is from the run of 2026-11-10 (records/cell_npu.json; its answers are in records/npu_timed_run1.json), on driver 32.0.100.4841. In the speed runs below, on driver 32.0.100.5540, every answer had the same tokens (new_ids) as in that run.

Speed over those 20 questions and one more short prompt (21 prompts of 23–40 tokens, answers of 2–27 tokens): with the settings above, 35 ms to the first token, then 73.5 tokens/s; with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone, 35 ms, then 63.9 tokens/s; without either setting, 154 ms, then 64.9 tokens/s (in the same measuring session; each the median of 3 runs).

Longer prompts take longer to the first token with NPUW_LLM_PREFILL_CHUNK_SIZE=64: 227 ms at 512 tokens and 493 ms at 1,000 tokens with the settings above (232 ms and 510 ms with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone; without either setting: 155 ms and 155 ms; median of 3 tries each, records/2026-11-12-best-perf/npu_long_prompt*.json).

Long prompts give the same answers with NPUW_LLM_PREFILL_CHUNK_SIZE=64 as without it: on 107 of 108 prompts longer than one 64-token chunk (65–1,000 tokens, answers of up to 48 tokens), the greedy answer had the same tokens in two runs with the setting and two without. The other one is the same answer worded differently: to "What is the capital of France? Answer with one word." at the end of a 65-token prompt, the answer is "France's capital is Paris." with the setting and "Paris" without it (records/2026-11-12/npu_chunk64_answers.json). Adding GENERATE_HINT="BEST_PERF" changed no answer: in two runs with it, all 130 prompts of that check (the 108 above and 22 shorter ones) gave the same tokens as with NPUW_LLM_PREFILL_CHUNK_SIZE=64 alone (records/2026-11-12-best-perf/npu_best_perf_answers.json).

Speed still being tuned (updates follow). Records: records/ (the speed runs above are in records/2026-11-12-best-perf/; tok_per_s in the run files is over whole answers, first token included).