Qwen3-1.7B-int8-sym-ov

Qwen/Qwen3-1.7B (revision 70d244c) as an OpenVINO IR with int8 symmetric per-channel weights, for the Intel NPU.

pip install "openvino-genai==2026.4.1.0" huggingface_hub
hf download mlboydaisuke/Qwen3-1.7B-int8-sym-ov --local-dir Qwen3-1.7B-int8-sym-ov

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("Qwen3-1.7B-int8-sym-ov", "NPU", NPUW_LLM_PREFILL_CHUNK_SIZE=64, CACHE_DIR="npu_cache")
print(pipe.generate("What is the capital of France? /no_think", max_new_tokens=32))

NPUW_LLM_PREFILL_CHUNK_SIZE=64 runs the prompt in chunks of 64 tokens instead of one pass padded to 1,024 tokens. The first token then comes sooner for a short prompt and later for a long one (numbers below). CACHE_DIR keeps the compiled model, so later loads skip the compile. Without these settings the pipeline runs with OpenVINO GenAI's NPU defaults.

Measured on one laptop's NPU

Intel Core Ultra 9 386H, Windows 11, Intel NPU driver 32.0.100.5540, OpenVINO GenAI 2026.4.1, MSI Center's Balanced mode, 2026-11-12:

Answer agreement: the first line of this export's answer matched Qwen/Qwen3-1.7B's (fp32, transformers, CPU) on 18 of 20 short questions (thinking off). The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. The count is from the run of 2026-11-10 (records/cell_npu.json; its answers are in records/npu_timed_run1.json), on driver 32.0.100.4841. In the speed runs below, on driver 32.0.100.5540, every answer had the same tokens (new_ids) as in that run.
Speed over those 20 questions and one more short prompt (21 prompts of 23–40 tokens, answers of 2–48 tokens): with the settings above, 53 ms to the first token, then 30.9 tokens/s; without NPUW_LLM_PREFILL_CHUNK_SIZE, in the same measuring session, 295 ms, then 30.5 tokens/s (each the median of 3 runs).
Longer prompts take longer to the first token with NPUW_LLM_PREFILL_CHUNK_SIZE=64: 365 ms at 512 tokens and 761 ms at 1,000 tokens (without it: 297 ms and 290 ms; median of 3 tries each, records/2026-11-12/npu_long_prompt*.json).
Long prompts give the same answers with NPUW_LLM_PREFILL_CHUNK_SIZE=64 as without it: on 107 of 108 prompts longer than one 64-token chunk (65–1,000 tokens, answers of up to 48 tokens), the greedy answer had the same tokens in two runs with the setting and two without. The other one is the same answer worded differently: to "Is 17 a prime number? Answer yes or no." at the end of a 512-token prompt, the answer begins "Yes, 17 is a prime number." with the setting and "Yes." then "17 is a prime number." without it (records/2026-11-12/npu_chunk64_answers.json).

Speed still being tuned (updates follow). Records: records/ (the speed runs above are in records/2026-11-12/; tok_per_s in the run files is over whole answers, first token included).

Exported with optimum-cli export openvino --task text-generation-with-past --weight-format int8 --sym (optimum-intel 2.2.0, nncf 3.4.0, openvino 2026.4.1) from the files of Qwen/Qwen3-1.7B at revision 70d244c, each checked against the Hub's sha256 (git blob sha1 for the small files).

加载中…

Qwen3-1.7B-int8-sym-ov

Qwen/Qwen3-1.7B (revision 70d244c) as an OpenVINO IR with int8 symmetric per-channel weights, for the Intel NPU.

pip install "openvino-genai==2026.4.1.0" huggingface_hub hf download mlboydaisuke/Qwen3-1.7B-int8-sym-ov --local-dir Qwen3-1.7B-int8-sym-ov

import openvino_genai as ov_genai pipe = ov_genai.LLMPipeline("Qwen3-1.7B-int8-sym-ov", "NPU", NPUW_LLM_PREFILL_CHUNK_SIZE=64, CACHE_DIR="npu_cache") print(pipe.generate("What is the capital of France? /no_think", max_new_tokens=32))

Measured on one laptop's NPU

Intel Core Ultra 9 386H, Windows 11, Intel NPU driver 32.0.100.5540, OpenVINO GenAI 2026.4.1, MSI Center's Balanced mode, 2026-11-12:

Answer agreement: the first line of this export's answer matched Qwen/Qwen3-1.7B's (fp32, transformers, CPU) on 18 of 20 short questions (thinking off). The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. The count is from the run of 2026-11-10 (records/cell_npu.json; its answers are in records/npu_timed_run1.json), on driver 32.0.100.4841. In the speed runs below, on driver 32.0.100.5540, every answer had the same tokens (new_ids) as in that run.

Speed over those 20 questions and one more short prompt (21 prompts of 23–40 tokens, answers of 2–48 tokens): with the settings above, 53 ms to the first token, then 30.9 tokens/s; without NPUW_LLM_PREFILL_CHUNK_SIZE, in the same measuring session, 295 ms, then 30.5 tokens/s (each the median of 3 runs).

Longer prompts take longer to the first token with NPUW_LLM_PREFILL_CHUNK_SIZE=64: 365 ms at 512 tokens and 761 ms at 1,000 tokens (without it: 297 ms and 290 ms; median of 3 tries each, records/2026-11-12/npu_long_prompt*.json).

Long prompts give the same answers with NPUW_LLM_PREFILL_CHUNK_SIZE=64 as without it: on 107 of 108 prompts longer than one 64-token chunk (65–1,000 tokens, answers of up to 48 tokens), the greedy answer had the same tokens in two runs with the setting and two without. The other one is the same answer worded differently: to "Is 17 a prime number? Answer yes or no." at the end of a 512-token prompt, the answer begins "Yes, 17 is a prime number." with the setting and "Yes." then "17 is a prime number." without it (records/2026-11-12/npu_chunk64_answers.json).

Speed still being tuned (updates follow). Records: records/ (the speed runs above are in records/2026-11-12/; tok_per_s in the run files is over whole answers, first token included).