bge-reranker-v2-m3-int4-ov

Model creator: BAAI.
Original model: BAAI/bge-reranker-v2-m3.
Source revision: 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e.

Description

OpenVINO IR export of the multilingual BGE reranker v2 M3 with int4 weights, prepared with Optimum Intel. It is an XLMRobertaForSequenceClassification cross-encoder: each query/document pair produces one relevance logit. Higher scores indicate greater relevance. It is not an embedding model, a generative reranker, or a two-class classifier.

Use it to reorder a candidate set after first-stage retrieval. Tokenize the query and document as a pair; let the tokenizer add the XLM-R special tokens. No custom prompt or manual separator concatenation is required.

Weight compression

NNCF weight-only compression: INT4_ASYM.

{
  "bits": 4,
  "sym": false,
  "group_size": 16,
  "ratio": 0.5,
  "all_layers": false,
  "backup_precision": "int8_asym"
}

No calibration dataset, AWQ or GPTQ is used. Activations are not statically quantized by this recipe; execution precision depends on the runtime and device.

INT4 denotes mixed-precision weight compression, not four bits for every tensor. Embedding/final layers are excluded from INT4 by all_layers=False; backup precision is INT8_ASYM. ratio is the requested INT4 fraction of eligible weight parameters, not a fraction of layers, a byte-size ratio or an accuracy guarantee.

Running model inference with Optimum Intel

python -m pip install "optimum-intel[openvino]==2.2.0" "transformers==4.57.6" sentencepiece

import torch
from optimum.intel import OVModelForSequenceClassification
from transformers import AutoTokenizer

model_id = "OpenVINO/bge-reranker-v2-m3-int4-ov"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForSequenceClassification.from_pretrained(model_id, device="CPU")

query = "What is the capital of France?"
documents = [
    "Berlin is the capital of Germany.",
    "Paris is the capital of France.",
    "A banana is a fruit.",
]
inputs = tokenizer(
    [query] * len(documents), documents,
    padding=True, truncation=True, max_length=512, return_tensors="pt",
)
logits = model(**inputs).logits.reshape(-1).float()
normalized_scores = torch.sigmoid(logits)
for index in torch.argsort(logits, descending=True).tolist():
    print(index, float(logits[index]), float(normalized_scores[index]), documents[index])

logits has shape [number_of_pairs, 1] before flattening. Sigmoid optionally maps each logit into [0, 1] and preserves ranking; it does not produce a calibrated probability. Do not apply a softmax across the single output or across unrelated documents.

Running model inference with OpenVINO GenAI

python -m pip install "openvino-genai==2026.4.1" huggingface-hub

import openvino_genai as genai
from huggingface_hub import snapshot_download

model_dir = snapshot_download(repo_id="OpenVINO/bge-reranker-v2-m3-int4-ov")
query = "What is the capital of France?"
documents = [
    "Berlin is the capital of Germany.",
    "Paris is the capital of France.",
    "A banana is a fruit.",
]
config = genai.TextRerankPipeline.Config()
config.top_n = len(documents)
config.max_length = 512
pipeline = genai.TextRerankPipeline(model_dir, "CPU", config)

ranked = pipeline.rerank(query, documents)
for index, score in ranked:
    print(index, score, documents[index])

GenAI loads the model and tokenizer IRs, tokenizes the query/document pairs, and returns (original_document_index, relevance_score) entries in descending score order. Scores are not calibrated probabilities. For an existing local export, use its directory as model_dir instead of calling snapshot_download.

Conversion

Using convert_models.py from the accompanying conversion bundle:

python convert_models.py --model bge-reranker-v2-m3 --precision int4

The script pins the source snapshot (including tokenizer files), explicitly selects the Transformers library and text-classification task, exports the encoder and OpenVINO tokenizer, applies the weight-compression recipe, and records package versions. It does not upload or run inference/accuracy benchmarks.

Input length

max_length=512 is the examples' truncation setting, not a 512-token model limit. It applies to the complete query/document pair, including special tokens, rather than separately to each string. The source model card's Transformers example also uses 512 as an example setting.
The pinned source configuration declares max_position_embeddings=8194 and pad_token_id=1. With XLM-R's position-ID offset, this corresponds to an 8192-token position budget, not an 8194-token input limit.

Related notebooks

Legal information

The source checkpoint is identified as Apache-2.0 in its pinned model card. Do not inherit the MIT license of older bge-reranker-base exports for this model. Refer to the original model for attribution, limitations, and citation. See also Intel's Global Human Rights Principles.

Loading…