embeddinggemma-2-text-int8-cw-onnx

The text model of EmbeddingGemma 2 as ONNX, fixed to run correctly on OpenVINO and compressed to INT8 symmetric, per-channel (274 MB, down from 1084 MB) with no measurable quality loss. Built for the Intel NPU.

It starts from onnx-community/embeddinggemma-2-ONNX, which has two problems on OpenVINO:

1. Wrong output. OpenVINO 2026.4.1's ONNX frontend mis-converts com.microsoft.RotaryEmbedding when position_ids is the shape [1] offset form, which that export uses for all 48 rotary nodes. Embeddings land at cosine ~0.81 to the reference and SciFact nDCG@10 drops to 0.582. Tracked in karanshukla/vinoSearch#1.
2. Size. The fp32 text model is 1.08 GB.

How it was made

Each RotaryEmbedding node was rewritten as rotate-half with plain ONNX ops (Slice, Mul, Sub, Add, Concat). The rewrite only accepts the exporter's exact form (3D input, constant offset [0], caches covering every position, interleaved=0) and refuses anything else. Under ONNX Runtime the rewritten graph gives the same embeddings as the original (cosine 1.0000).

The weights were then compressed with NNCF 3.4.0, data-free, ONNX in and ONNX out:

nncf.compress_weights(model, mode=nncf.CompressWeightsMode.INT8_SYM)

All 147 weight layers, including the 140M-parameter embedding table, are INT8 symmetric per-channel. Nothing is group-quantized, which is the weight format the NPU compiles well. Staying ONNX means it loads in runtimes that ship only OpenVINO's ONNX frontend, such as the onnxruntime-openvino wheel's libraries.

Results

BEIR SciFact (test, 5,183 docs, 300 queries), documents truncated to 256 tokens. Measured on a Core 5 320 (Wildcat Lake NPU), OpenVINO 2026.4.1, NPU driver 1.35.0, 2026-10-18.

Cosine to the fp32 reference (ONNX Runtime on the original export): 0.9996 or better on OpenVINO CPU, OpenVINO NPU and ONNX Runtime. INT4 was left out: it saves 54 MB and costs 0.03 nDCG.

Usage

Inputs are input_ids and attention_mask. Text-only use feeds the three media inputs empty, shape [0, 512]. Use sentence_embedding and L2-normalise it. Prefix text the way the base model expects (see config_sentence_transformers.json):

import numpy as np
import openvino as ov
from tokenizers import Tokenizer

L = 256
tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(L)
tok.enable_padding(length=L)

core = ov.Core()
model = core.read_model("onnx/model.onnx")
model.reshape({
    "input_ids": [1, L], "attention_mask": [1, L],
    "image_features": [0, 512], "video_features": [0, 512], "audio_features": [0, 512],
})
compiled = core.compile_model(model, "NPU", {"CACHE_DIR": "ov-cache"})

enc = tok.encode("task: search result | query: what causes the northern lights")
empty = np.zeros((0, 512), np.float32)
out = compiled({
    "input_ids": np.array([enc.ids], np.int64),
    "attention_mask": np.array([enc.attention_mask], np.int64),
    "image_features": empty, "video_features": empty, "audio_features": empty,
})[compiled.output("sentence_embedding")][0]
embedding = out / np.linalg.norm(out)

onnx/model_text.onnx is the same model with the three media inputs baked in as empty constants, so its only inputs are input_ids and attention_mask. It shares model.onnx_data. Use it where a zero-size tensor cannot be built: OpenVINO's C API (and so its Rust bindings) refuses a 0 dimension. Its embeddings match model.onnx exactly on CPU and to cosine 0.9999 on the NPU.

onnx/model_text_q32.onnx and onnx/model_text_d256.onnx are model_text.onnx with its inputs fixed at [1, 32] (queries) and [1, 256] (documents), also sharing model.onnx_data. The NPU needs static shapes, and a graph that already has them can be compiled straight from its path (core.compile_model(path, "NPU")), which lets OpenVINO's model cache skip reading the weights: on a warm cache, 340 MB peak and 0.04 s instead of 630 MB and 0.17 s for read + reshape + compile. Their embeddings match model_text.onnx reshaped to the same shape exactly (cosine 1.000000 on the NPU). Made by bench/static_shapes.py.

NPU notes, measured on OpenVINO 2026.4.1:

• Use batch 1 on the NPU. A static batch above 1 mixes rows across the batch (cosine to CPU as low as 0.59 on other models at batch 8). Keep several requests in flight instead; it is correct and faster.
• Shapes must be static, as above. Padding to a fixed length does not change the embedding.

Images

onnx/model_image_280.onnx, onnx/model_image_140.onnx and onnx/model_image_70.onnx are model_text.onnx with image_features an input again, fixed at [tokens, 512]. Their input_ids and attention_mask are fixed at tokens + 4, rounded up to 16: [1, 288], [1, 144] and [1, 80]. They share model.onnx_data, so images add no text weights. They take the output of the static vision encoders in arankays/embeddinggemma-2-vision-static-onnx unsliced. The model merges the first N rows of image_features, N being the image tokens in input_ids, and ignores the rest.

The prompt is [2, 255999, 258880 × N, 258882, 1] (bos, begin-of-image, N image tokens, end-of-image, eos), padded at the end with id 0 and attention_mask 0. Pad with 0, not with extra image tokens: those are attended even when masked (cosine 0.9993 to unpadded). Made by bench/static_image_text.py.

These embeddings land at cosine 0.9994 or better to sentence-transformers' fp32 reference on the NPU, at 40–50 ms per image (170 ms on CPU at 280 tokens). Flickr30k retrieval, speed and energy are on the vision repo's card.

The audio encoder in onnx-community/embeddinggemma-2-ONNX is not included.

Used by vinoSearch.

License

Apache 2.0, same as the base model. EmbeddingGemma 2 is by Google DeepMind. Modified from onnx-community/embeddinggemma-2-ONNX: rotary nodes rewritten, weights compressed to INT8.