Gemma 4 E4B-it — every component on the Intel NPU

Now also on the shared engine onw (俺のNPUがこんなに動くわけない): one engine for LFM2, Qwen3.6-35B-A3B, Qwen3.5-9B and Gemma 4 — start.bat ryugyosoft/gemma-4-E4B-it-onw. This repo stays the standalone version.

google/gemma-4-E4B-it (text + image) running with all model components on the Intel NPU: vision encoder, text embeddings, the 2.7 GB per-layer embeddings, the 42-layer decoder and the LM head. The host only tokenizes, preprocesses pixels, builds attention masks, copies KV rows and samples. Comes with an OpenAI-compatible API server (streaming, image input).

This is not llama.cpp and not OpenVINO GenAI's pipeline: the model graphs were rewritten for the NPU and a small generation loop manages the KV cache (details below). NPU execution itself is OpenVINO.

Tested on two machines: Core Ultra 9 285HX (Arrow Lake HX, NPU 3720), Windows 11, NPU driver 32.0.100.5540, OpenVINO 2026.5.0b1 (nightly); and Core Ultra 5 226V (Lunar Lake, NPU 4000), Ubuntu 26.04, Linux NPU driver 1.38.0, OpenVINO 2026.5.0 nightly (decode 8.2-8.5 tok/s, vision 1.0-1.4 s, prefill 3.2-3.7 s for ~290 tokens). Other NPUs are untested.

$ python gemma4_npu.py cat.jpg "この画像に何が写っていますか？"
[vision] NPU (NPUW)
[load] 14.8s  devices={'vision': 'NPU', 'emb': 'NPU', 'ple': 'NPU', 'lm': 'NPU'}
この画像には、大きな段ボール箱の中に横たわっている灰色の縞模様の猫が写っています。猫はリラックスしている
様子で、箱の縁に頭を乗せて目を閉じているかのように見えます。背景は明るい部屋で、白いラグの上に箱が置かれています。
[perf] vision 2006 ms | prefill 288 tok 4160 ms | decode 63 tok 6.6 tok/s

First start compiles everything for the NPU (~6 min, +3.5 min for the 64-token block); later starts load from models/npu_cache in ~20 s.

New in v2

The previous version is kept as the v1 tag: hf download ryugyosoft/gemma-4-E4B-it-npu --revision v1 --local-dir .... (all exact: the output is the same as plain greedy decoding)

• Prompt lookup decoding (greedy requests, i.e. temperature 0, the UI default): when the last tokens also occur earlier in the conversation, the tokens that followed there are proposed as a draft and checked in one 16-token decoder pass; the accepted ones cost nothing extra. No draft model, no extra memory. Code edits 7.6 → 13.8 tok/s, "summarize using the original wording" 7.4 → 15.1 tok/s, free chat unchanged.
• Prefix reuse: the KV cache of the previous turn is kept; a follow-up only prefills what changed (turn 2 of an image chat: 4.9 s → 1.7 s, 299 of 319 tokens reused). Reported as usage.prompt_tokens_details.cached_tokens.
• 64-token prefill blocks (4.5 ms/token vs 12.7 with 16-token blocks). The extra decoder variant holds another ~2 GB, so it is used only with ≥ 24 GB RAM; G4_S64=1 / G4_S64=0 forces it on / off.

Quick start (one click)

1. Download this repo: hf download ryugyosoft/gemma-4-E4B-it-npu --local-dir gemma4-npu (or "Clone repository" on this page). Python 3.10+ must be installed.
2. Start it:
   • Windows: double-click start.bat
   • Ubuntu: bash start.sh
3. A browser opens the chat UI at http://localhost:8000/ (text + images: attach, paste or drop).

The first start installs packages into .venv (~2 GB) and compiles the model for the NPU (6-10 min); later starts take ~20 s. Without an NPU (or its driver) it falls back to the CPU automatically. The same server also speaks the OpenAI API at http://localhost:8000/v1. Options go after the script: start.bat --fast, bash start.sh --port 8080 --host 0.0.0.0.

Manual setup

hf download ryugyosoft/gemma-4-E4B-it-npu --local-dir gemma4-npu
cd gemma4-npu
python -m venv .venv && .venv\Scripts\activate
pip install --pre -r requirements.txt --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly --extra-index-url https://download.pytorch.org/whl/cpu

python gemma4_npu.py photo.jpg "この画像を説明して"     # CLI, image + prompt
python gemma4_npu.py - "Explain photosynthesis briefly."  # text only
python server.py --port 8000 --open                       # chat UI + OpenAI-compatible API (add --fast for the faster decoder)
python example_client.py photo.jpg                        # official OpenAI SDK against the server

Ubuntu

The IRs are OS-independent; the code runs unchanged on Linux (NPU: Ubuntu 26.04 on Lunar Lake / NPU 4000; CPU mode: Ubuntu 22.04 / Python 3.10). For the NPU you need Intel's Linux NPU driver:

1. Install the latest release from https://github.com/intel/linux-npu-driver/releases (v1.38.0 targets Ubuntu 24.04 / 26.04; follow the release notes: libtbb12, the driver .debs, and the Level Zero loader libze1).
2. Give your user access to the device and check it is there:

   sudo gpasswd -a ${USER} render && newgrp render
   ls /dev/accel/accel0
   python -c "import openvino as ov; print(ov.Core().available_devices)"   # should list NPU
   

3. Then the same commands as above (python3 -m venv .venv && source .venv/bin/activate, ...).

Without an NPU driver everything still runs on the CPU: python server.py --device CPU, or python gemma4_npu.py photo.jpg "..." --vision-device CPU --emb-device CPU --ple-device CPU --lm-device CPU.

API

GET /v1/models, POST /v1/chat/completions with stream, stream_options.include_usage, max_tokens / max_completion_tokens, temperature, top_p, seed, stop, system / multi-turn messages, and image_url parts (data: URLs or http(s)). Requests are served one at a time.

from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
r = c.chat.completions.create(model="gemma-4-e4b-it-npu", messages=[{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
    {"type": "text", "text": "What is in this image?"}]}])

Options: --soft-tokens 70|140|280 (image token budget: fewer = faster vision, less detail), --fast.

Limitations

• Context is a static 1024-token KV cache (one image at 280 soft tokens ≈ 290 prompt tokens).
• One request at a time; no batching. Prefix reuse keeps only the most recent conversation (a different chat starts from scratch).
• Audio input is not supported (the audio encoder is not included).
• Needs a pre-release OpenVINO (2026.5 nightly). NPU tested on Windows (NPU 3720) and Ubuntu 26.04 (NPU 4000).
• On NPU 4000 the static decoder must be compiled through NPUW (per-layer folding): compiled as one graph it does not finish (>30 min). gemma4_npu.py selects this from the NPU architecture; G4_LM_CONFIG (JSON) overrides it.
• The NPU workarounds (head_dim-512 attention, per-process memory split) were found on NPU 3720. Other NPU generations or the Linux driver may behave differently; layer_probe.py compares each decoder layer on NPU vs CPU if outputs look wrong.
• --fast is faster but visibly less accurate; use the default for anything factual.

What is in this repo

How it works (what had to be fixed for NPU 3720)

Vision: static shapes. The original OpenVINO vision IR is data-dependent in three places.

1. Patch position embeddings were OneHot(pos, 10240) @ table (~79 GFLOP, 1.6 s on the NPU). Rewritten as the equivalent row lookup; now a few ms.
2. The pooler's one-hot depth was a runtime scalar; it always equals the soft-token budget.
3. NonZero -> GatherND dropped padded tokens. The norm and projection after it are per-token, so all tokens are computed and a valid mask is returned; the host slices. Matches the original on CPU (cosine 0.998).

Decoder: head_dim-512 attention is wrong on NPU 3720. Through OpenVINO's NPUW LLM mode the model produced garbage from the first token, for every quantization and option tried, while the same partitioning on NPUW_DEVICES=CPU was correct. Comparing each layer against CPU (layer_probe.py) showed errors only at the 7 full-attention layers (5, 11, ..., 41), and inside them only at the attention output (~100% error) with q/k/v, KV concat, GQA broadcast and mask all correct. The broken piece is the P·V product with a 512-wide output. Since attention is linear in V, the fix keeps one SDPA per layer but doubles its heads so each head's V is 256 wide (npu_fixes.split_sdpa_v); error drops to 0.57%. NPUW LLM mode rejects that graph, so make_static_lm.py builds static decoders instead, with host-built masks and a host-managed fp16 KV cache kept in the NPU request's own input buffers (static_lm.py). The decoder also needs symmetric INT4/INT8 weights: NPU 3720 cannot dequantize the official asymmetric INT4.

Per-layer embeddings: NPU memory per process. NPU 3720 cannot keep the whole 2.7 GB int8 table resident in one process next to the decoders (BufferInfo already exists at load), but the limit is per process: 2 chunks live in the main process and 2 in a worker, computed concurrently and summed by a tiny NPU Add model.

Speed (per generated token, default decoder): decoder 109 ms, LM head 20 ms, per-layer embeddings 15 ms, text embedding 1 ms. From 3.4 tok/s: returning only the new K/V rows in fp16 (3.9), a dedicated 1-token decoder instead of padded 16-token blocks (6.1), KV kept in NPU buffers shared by both decoders (6.4), overlapped per-layer embeddings (6.6), channel-wise INT4 for --fast (8.3).

Rebuilding the IRs

hf download OpenVINO/gemma-4-E4B-it-int4-ov --local-dir models/gemma-4-E4B-it-int4-ov
python split_vision.py 280 && python split_vision.py 140 && python split_vision.py 70
python split_ple.py 4 32
pip install nncf "git+https://github.com/huggingface/optimum-intel.git"
optimum-cli export openvino -m google/gemma-4-E4B-it --task image-text-to-text --weight-format int4 --sym --group-size 128 --ratio 1.0 --backup-precision int8_sym gemma-4-E4B-it-int4-allsym-ov
python make_static_lm.py gemma-4-E4B-it-int4-allsym-ov models/lm_static 64 1024
python make_static_lm.py gemma-4-E4B-it-int4-allsym-ov models/lm_static 16 1024
python make_static_lm.py gemma-4-E4B-it-int4-allsym-ov models/lm_static 1 1024
# --fast: same export with --group-size -1, into models/lm_static_cw

日本語での概要

Gemma 4 E4B-it（画像＋テキスト）の 全部品を Intel NPU 上で動かす 一式です。Vision・埋め込み・デコーダ・ 出力層がすべてNPUで動きます。Windowsは start.bat をダブルクリック、Ubuntuは bash start.sh で、 ブラウザのチャット画面（画像の添付・貼り付け・ドロップ対応）が開きます。OpenAI互換APIとしても使えます。 NPU 3720（Core Ultra 9 285HX）で検証。生成速度は 6.6-7.6 tok/s（--fast で 8.3 tok/s、ただし事実の正確さが落ちます）。 v2 では、会話中の文字列を再利用する回答（コード修正・要約など）を 13-15 tok/s に速くするプロンプトルックアップ デコーディング、2ターン目以降は差分だけを処理するプレフィックス再利用、64トークン単位のプリフィル（RAM 24GB以上） を追加しました。いずれも出力は通常のグリーディ生成と同一です。 NPU 3720では全体アテンション層（ヘッド次元512）の出力計算が壊れるため、ヘッドを倍にしてVを256幅に割る書き換えで 回避しています。詳細は上の "How it works" を参照してください。

License

Apache 2.0, same as the base model google/gemma-4-E4B-it. The weights here are converted/re-quantized from it (and the text embeddings from OpenVINO/gemma-4-E4B-it-int4-ov).