BAAI/bge-reranker-v2-m3 - OpenVINO int8

OpenVINO int8 IR of BAAI/bge-reranker-v2-m3 (XLM-RoBERTa-large cross-encoder), quantized for CPU inference as the relevance reranker in a single-engine semantic grounding pipeline. The int8 preserves the fp32 max-over-chunks grounding signal at pearson 0.9976 - no measurable quality loss.

Model

• Base model - BAAI/bge-reranker-v2-m3 (XLM-RoBERTa-large cross-encoder)
• Format - OpenVINO IR (openvino_model.xml + openvino_model.bin), int8
• Quantization - NNCF int8 (Fast Bias Correction)
• int8 parity vs fp32 - pearson 0.9976 on the max-over-chunks grounding score
• Size - 571 MB int8 (fp32 > 2 GB)
• Role - relevance reranker
• Runtime - OpenVINO on CPU (x86-64 Intel/AMD via AVX2 / AVX-512-VNNI; ARM/Graviton via the OpenVINO ARM plugin)

Usage

from huggingface_hub import snapshot_download
import numpy as np, openvino as ov
from transformers import AutoTokenizer

d = snapshot_download("stellars/bge-reranker-v2-m3-openvino-int8")
core = ov.Core()
model = core.compile_model(core.read_model(f"{d}/openvino_model.xml"), "CPU")
tok = AutoTokenizer.from_pretrained(d)

enc = tok(["a sentence"], ["another sentence"], return_tensors="np",
          padding=True, truncation=True, max_length=512)
feed = {"input_ids": enc["input_ids"].astype(np.int64),
        "attention_mask": enc["attention_mask"].astype(np.int64)}
logits = model(feed)[model.output(0)]

Quantization details

• Method - NNCF int8 (Fast Bias Correction), via NNCF / optimum-intel
• Calibration - 128 (claim, chunk) pairs sampled from 3 selected agentic chat RAG datasets
• Why this matters - DeBERTa-v2 disentangled attention is int8-hostile (plain dynamic int8 collapses to pearson 0.35); NNCF SmoothQuant migrates the activation outliers into the per-channel weights, which is the only int8 method that preserves the signal. The standard-attention bge models quantize cleanly with plain NNCF int8

License

Inherits the apache-2.0 license of the base model.