SigLIP 2 So400m (patch16-512) — OpenVINO IR (INT8 / FP16, dual-tower)

OpenVINO™ Intermediate Representation of google/siglip2-so400m-patch16-512, exported with optimum-intel and quantized with NNCF (INT8 weight-only, data-free).

• Vision tower 429M + text tower 708M parameters, 27 + 27 layers, hidden 1152, 512×512 input, patch 16 (1024 tokens).
• Ready for Intel CPU (AVX2 / AVX-VNNI / AVX-512) and Intel NPU (/dev/accel/accel0, Core Ultra).
• Verified against the PyTorch checkpoint: fp16 logits cosine 1.0000, INT8 logits cosine 0.9997 (Top-1 unchanged).

Repository layout

Shared at the repo root: config.json, preprocessor_config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, chat_template.jinja, logit_head.json, demo.py, coco.jpg (example image).

int8_split/ holds the two towers separately, which is the fastest deployment shape: the text tower runs once per label set (cache the embeddings), and only the vision tower runs per image. Its text tower is exported with a static batch of 10 (input_ids [10, 64]) so it compiles on NPU; pad shorter label lists and slice the embeddings back (see demo.py).

Model inputs / outputs

int8/openvino_model.xml and fp16/openvino_model.xml:

inputs : input_ids    int64   [batch_labels, seq_len]     # padding to seq_len=64, no attention_mask!
         pixel_values float32 [1, 3, 512, 512]
outputs: logits_per_image float32 [1, batch_labels]
         logits_per_text  float32 [batch_labels, 1]
         image_embeds     float32 [1, 1152]               # L2-normalised
         text_embeds      float32 [batch_labels, 1152]    # L2-normalised

int8_split/: vision_model.xml pixel_values → image_embeds, text_model.xml input_ids → text_embeds.

Preprocessing is the original SigLIP processor: resize to 512×512, rescale 1/255, normalise with mean = std = 0.5. The attention mask is baked into the exported graph (all ones) — always pad the text to a fixed length (64) and do not pass an attention_mask.

Similarity head (needed when you use the split towers): the embeddings are already L2-normalised and

logits = exp(logit_scale) * (image_embeds @ text_embeds.T) + logit_bias
       = 109.88801574707031 * cos_sim - 15.932648            # see logit_head.json

The checkpoint stores logit_scale as a log value (4.6996384), hence exp(). The combined IR already includes both terms in logits_per_image — softmax over it for single-label classification, or sigmoid for multi-label / retrieval scoring.

Usage

optimum-intel (convenience)

from PIL import Image
from transformers import AutoProcessor
from optimum.intel import OVModelForZeroShotImageClassification

processor = AutoProcessor.from_pretrained("YOUR_REPO_ID")
model = OVModelForZeroShotImageClassification.from_pretrained("YOUR_REPO_ID/int8")   # or /fp16

image = Image.open("coco.jpg").convert("RGB")
labels = ["cat", "dog", "wolf"]
inputs = processor(text=[f"This is a photo of a {l}" for l in labels], images=[image],
                   padding="max_length", max_length=64, truncation=True, return_tensors="np")

logits = model(**inputs).logits_per_image            # [1, 3]
print(logits)

OpenVINO runtime (what the benchmarks below use)

import numpy as np, openvino as ov
from PIL import Image
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("YOUR_REPO_ID")
labels = ["cat", "dog", "wolf", "tiger"]
inputs = processor(text=[f"This is a photo of a {l}" for l in labels],
                   images=[Image.open("coco.jpg").convert("RGB")],
                   padding="max_length", max_length=64, truncation=True, return_tensors="np")

core = ov.Core()
model = core.read_model("int8/openvino_model.xml")
# NPU needs fully static shapes:
# model.reshape({"input_ids": inputs["input_ids"].shape, "pixel_values": inputs["pixel_values"].shape})
compiled = core.compile_model(model, "CPU")          # or "NPU", "GPU"

out = compiled({"input_ids": np.asarray(inputs["input_ids"], dtype=np.int64),
                "pixel_values": np.asarray(inputs["pixel_values"], dtype=np.float32)})
logits = np.asarray(out["logits_per_image"])
p = np.exp(logits - logits.max(axis=-1, keepdims=True)); p /= p.sum(axis=-1, keepdims=True)
print(sorted(zip(labels, p[0]), key=lambda x: -x[1])[:3])

Dual tower with cached text embeddings (fastest per image)

import json, numpy as np, openvino as ov

core = ov.Core()
vision = core.compile_model("int8_split/vision_model.xml", "NPU")
text = core.compile_model("int8_split/text_model.xml", "NPU")   # static batch = 10

# once per label set (pad to 10 if you have fewer labels, then slice [:n])
text_embeds = text({"input_ids": ids_padded})[0][:len(labels)]
image_embeds = vision({"pixel_values": pixel_values})[0]

head = json.load(open("logit_head.json"))
logits = head["logit_scale_exp"] * (image_embeds @ text_embeds.T) + head["logit_bias"]

demo.py in this repo implements both paths (--device CPU|NPU, --ir, --split-dir):

python demo.py --image coco.jpg --labels "cat,dog,wolf,tiger" --device CPU
python demo.py --image coco.jpg --labels "cat,dog,wolf,tiger" --device NPU
python demo.py --image coco.jpg --labels "cat,dog,wolf,tiger" --device CPU --split-dir ""   # combined IR

Benchmarks

Intel Core Ultra 5 125H (Meteor Lake, 14C/18T, no AVX-512 FP16), 30 GiB RAM, OpenVINO 2026.2.1, PyTorch 2.12.0+cpu. Input: one 512×512 image + 10 labels padded to 64 tokens; mean of 10–15 iterations after 3 warm-ups, fully static shapes, PERFORMANCE_HINT=LATENCY.

• fp16 is slower than fp32 on this CPU: without AVX-512 FP16 the fp16 weights are up-converted per inference. Use int8/ (or fp16/ on hardware with native FP16 support) for CPU.
• CPU throughput with 4 parallel infer requests: ≈ 0.40 images/s (2.5 s/image amortised); the CPU is compute/bandwidth bound, so concurrency buys little. NPU is the step change (690 ms/image).
• With the split towers and cached text embeddings the per-image cost is the vision tower alone (NPU 428 ms, CPU INT8 2124 ms).

Accuracy (vs the PyTorch fp32 checkpoint, same inputs)

INT8 shifts individual logits by up to ≈1.0 in absolute value (the head scale is ≈110, so it amplifies cosine-level noise), but probabilities and ranking are preserved. Use fp16/ when you need bit-closer agreement with the original checkpoint.

How it was converted

# fp16 IR (bit-identical to fp16/ in this repo).
# USE_TF=0 works around an OpenVINO 2026.2.x bug: its TF-frontend probe calls
# exec("del tensorflow") and raises NameError whenever tensorflow is importable.
# (Equivalent Python API: optimum.exporters.openvino.main_export(..., ov_config=OVConfig(dtype="fp16")).)
USE_TF=0 optimum-cli export openvino --model google/siglip2-so400m-patch16-512 \
    --task zero-shot-image-classification --library transformers --weight-format fp16 fp16/

# INT8 weight-only (data-free), per channel, symmetric — activations stay fp16/fp32
python -c "
import nncf, openvino as ov
m = ov.Core().read_model('fp16/openvino_model.xml')
ov.save_model(nncf.compress_weights(m, mode=nncf.CompressWeightsMode.INT8_SYM, ratio=1.0),
              'int8/openvino_model.xml', compress_to_fp16=True)"

The split towers were traced from the checkpoint (get_image_features / get_text_features) and saved as two standalone IRs with clean port names. --weight-format fp16 also matters because optimum auto-applies an int8_asym weight compression to ≥1B-parameter models when no weight format is given.

Notes and limitations

• No attention_mask input — it is baked in as all ones; pad text to a fixed length (64).
• int8_split/ text tower is static batch 10 (NPU-friendly). For arbitrary label counts use int8/openvino_model.xml (dynamic batch) or re-export the text tower with a dynamic first dimension.
• NPU: compile with fully static shapes; if a compile/inference fails, the NPU driver may stay in a broken state for that process (ZE_RESULT_ERROR_UNINITIALIZED) — run one model per process.
• transformers may warn about an "incorrect regex pattern" in tokenizer_config.json. This file is an unmodified copy of the base checkpoint's tokenizer, so tokenization matches the original model (fix_mistral_regex=True silences the warning at load time).
• These IRs are derived artifacts: license follows the base model, Apache-2.0 (google/siglip2-so400m-patch16-512).