gemma-4-E2B-it-int4-cw-ov

gemma-4-E2B-it in OpenVINO IR with the language model compressed to INT4 symmetric, channel-wise (group_size=-1), the weight format the Intel NPU compiles efficiently.

The official OpenVINO/gemma-4-E2B-it-int4-ov uses asymmetric group-128 INT4. On an Intel NPU (NPU5020, OpenVINO 2026.4.1) its compile unrolls each decoder block from ~270 ops into ~158,000 and does not finish. This export compiles block by block in well under a second each (about 2.5 min for the whole first compile, seconds once cached with CACHE_DIR).

How it was made

The language model was taken from OpenVINO/gemma-4-E2B-it-fp16-ov and compressed with NNCF 3.4.0:

nncf.compress_weights(
    model,
    mode=nncf.CompressWeightsMode.INT4_SYM,
    group_size=-1,
    ratio=0.8,
    sensitivity_metric=nncf.SensitivityMetric.WEIGHT_QUANTIZATION_ERROR,
    backup_mode=nncf.BackupMode.INT8_SYM,
)

165 of 279 weight layers are INT4 sym per-channel and the 111 most sensitive (plus the LM head) are INT8 sym per-channel. All-INT4 (ratio=1.0) garbled some short outputs; asymmetric INT8 backup was ~10x slower to decode on the NPU. Everything stays per-channel and symmetric, so nothing is group-quantized. Embeddings, vision encoder and tokenizers are unchanged from OpenVINO/gemma-4-E2B-it-int4-ov.

Usage (OpenVINO GenAI 2026.4+)

import openvino_genai as ov_genai

pipe = ov_genai.VLMPipeline(model_dir, "NPU", CACHE_DIR="ov-cache")
cfg = pipe.get_generation_config()
cfg.max_new_tokens = 200
cfg.min_new_tokens = 2  # NPU single-step NaN workaround (openvino#35222)
print(pipe.generate("Say hi in five words.", generation_config=cfg))

Used by vinoPop.

License

Apache 2.0, same as the base model. Gemma 4 is by Google DeepMind.