bge-m3-int4-ov

Model creator: BAAI.
Original model: BAAI/bge-m3.
Source revision: 5617a9f61b028005a4858fdac845db406aefb181.

Description

OpenVINO IR export of the multilingual BGE-M3 encoder with int4 weights, prepared with Optimum Intel. This artifact supports dense retrieval only. Sparse lexical weights and ColBERT multi-vector projections are not exported.

The IR returns last_hidden_state with shape [batch, tokens, 1024], not ready-made sentence vectors. Apply CLS pooling followed by L2 normalization to obtain 1024-dimensional dense embeddings. BGE-M3 does not require the query instruction prefix used by some earlier BGE models.

Weight compression

NNCF weight-only compression: INT4_ASYM.

{
  "bits": 4,
  "sym": false,
  "group_size": 32,
  "ratio": 1.0,
  "all_layers": false,
  "backup_precision": "int8_asym"
}

No calibration dataset, AWQ or GPTQ is used. Activations are not statically quantized by this recipe; execution precision depends on the runtime and device.

INT4 denotes mixed-precision weight compression, not four bits for every tensor. Embedding/final layers are excluded from INT4 by all_layers=False; backup precision is INT8_ASYM. ratio is the requested INT4 fraction of eligible weight parameters, not a fraction of layers, a byte-size ratio or an accuracy guarantee.

Running model inference with Optimum Intel

python -m pip install "optimum-intel[openvino]==2.2.0" "transformers==4.57.6" sentencepiece

import torch.nn.functional as F
from optimum.intel import OVModelForFeatureExtraction
from transformers import AutoTokenizer

model_id = "OpenVINO/bge-m3-int4-ov"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForFeatureExtraction.from_pretrained(model_id, device="CPU")

texts = [
    "What is BGE-M3?",
    "BGE-M3 is a multilingual text embedding model.",
    "The weather is sunny today.",
]
inputs = tokenizer(
    texts, padding=True, truncation=True, max_length=512, return_tensors="pt"
)
token_embeddings = model(**inputs).last_hidden_state
embeddings = F.normalize(token_embeddings[:, 0].float(), p=2, dim=1)
print(embeddings.shape)  # [3, 1024]
print(embeddings[0] @ embeddings[1:].T)  # Cosine similarities, not probabilities.

Apply the same pooling and normalization to queries and documents. Do not prepend an instruction. This is a Transformers feature-extraction export, not a packaged Sentence Transformers pipeline with built-in pooling.

Running model inference with OpenVINO GenAI

python -m pip install "openvino-genai==2026.4.1" huggingface-hub

import openvino_genai as genai
from huggingface_hub import snapshot_download

model_dir = snapshot_download(repo_id="OpenVINO/bge-m3-int4-ov")
config = genai.TextEmbeddingPipeline.Config()
config.pooling_type = genai.TextEmbeddingPipeline.PoolingType.CLS
config.normalize = True
config.max_length = 512
config.query_instruction = ""
config.embed_instruction = ""
pipeline = genai.TextEmbeddingPipeline(model_dir, "CPU", config)

query = "What is BGE-M3?"
documents = [
    "BGE-M3 is a multilingual text embedding model.",
    "The weather is sunny today.",
]
query_embedding = pipeline.embed_query(query)
document_embeddings = pipeline.embed_documents(documents)
similarities = [
    sum(q * d for q, d in zip(query_embedding, embedding))
    for embedding in document_embeddings
]
for index in sorted(range(len(documents)), key=lambda i: similarities[i], reverse=True):
    print(index, similarities[index], documents[index])

GenAI loads the model and tokenizer IRs from the downloaded directory and applies CLS pooling and L2 normalization. The dot products above are cosine similarities, not probabilities. For an existing local export, use its directory as model_dir instead of calling snapshot_download.

Conversion

Using convert_models.py from the accompanying conversion bundle:

python convert_models.py --model bge-m3 --precision int4

The script pins the source snapshot (including tokenizer files), explicitly selects the Transformers library, exports the encoder and OpenVINO tokenizer, applies the weight-compression recipe, and records package versions. It does not upload to Hugging Face or run inference/accuracy benchmarks.

Model properties and export scope

The original model card specifies a sequence length of 8192 tokens. The examples' max_length=512 is a demonstration truncation setting, not a 512-token model limit.
The source supports dense, sparse and ColBERT retrieval. This export contains only the dense encoder; the sparse and ColBERT projection heads are not included.

Related notebooks

Legal information

The source model is identified as MIT-licensed in its pinned model card. Refer to the original model for attribution, limitations, and citation. See also Intel's Global Human Rights Principles.

Loading…