bge-base-en-v1.5-int8-ov

• Model creator: BAAI
• Original model: bge-base-en-v1.5

Description

This is bge-base-en-v1.5 model converted to the OpenVINO™ IR (Intermediate Representation) format with quantization to INT8 by NNCF.

Disclaimer: Model is provided as a preview and may be update in the future.

Quantization Parameters

Weight compression was performed using nncf.compress_weights with the following parameters:

• mode: INT8_ASYM

For more information on quantization, check the OpenVINO model optimization guide.

Compatibility

The provided OpenVINO™ IR model is compatible with:

• OpenVINO version 2025.3.0 and higher
• Optimum Intel 1.25.2 and higher

Running Model Inference with Optimum Intel

1. Install packages required for using Optimum Intel integration with the OpenVINO backend:

pip install optimum[openvino]

2. Run model inference:

import torch
from transformers import AutoTokenizer

from optimum.intel.openvino import OVModelForFeatureExtraction


# Sentences we want sentence embeddings for
sentences = ["Sample Data-1", "Sample Data-2"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('OpenVINO/bge-base-en-v1.5-int8-ov')
model = OVModelForFeatureExtraction.from_pretrained('OpenVINO/bge-base-en-v1.5-int8-ov')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
model_output = model(**encoded_input)

# Perform pooling. In this case, cls pooling.
sentence_embeddings = model_output[0][:, 0]

# normalize embeddings
sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
print("Sentence embeddings:", sentence_embeddings)

For more examples and possible optimizations, refer to the Inference with Optimum Intel.

You can find more detailed usage examples in OpenVINO Notebooks:

• RAG text generation

Running Model with OpenAI client and OpenVINO Model Server

1a. Deploy model on Windows using binary package:

mkdir C:\models
ovms.exe --rest_port 8000 --source_model OpenVINO/bge-base-en-v1.5-int8-ov --model_repository_path C:\models

1b. Deploy model in a Docker container:

mkdir -p ${HOME}/models
export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models openvino/model_server:latest-gpu \
--rest_port 8000 --model_repository_path /models --source_model OpenVINO/bge-base-en-v1.5-int8-ov
2. Install the client library:

pip install openai "numpy<2"

3. Run the client:

from openai import OpenAI import numpy as np

client = OpenAI( base_url="http://localhost:8000/v1", api_key="unused" ) model = "OpenVINO/bge-base-en-v1.5-int8-ov" embedding_responses = client.embeddings.create( input=[ "That is a happy person", "That is a very happy person" ], model=model, ) embedding_from_string1 = np.array(embedding_responses.data[0].embedding) embedding_from_string2 = np.array(embedding_responses.data[1].embedding) cos_sim = np.dot(embedding_from_string1, embedding_from_string2)/(np.linalg.norm(embedding_from_string1)*np.linalg.norm(embedding_from_string2)) print("Similarity score as cos_sim", cos_sim)

## Limitations

Check the original [model card](https://huggingface.co/BAAI/bge-base-en-v1.5) for limitations.

## Legal information

The original model is distributed under [MIT](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE) license. More details can be found in [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5).

## Disclaimer

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.