Qwen3-Embedding-0.6B-int8-ov

• Model creator: Qwen
• Original model: Qwen3-Embedding-0.6B

Description

This is Qwen3-Embedding-0.6B model converted to the OpenVINO™ IR (Intermediate Representation) format with weights compressed to INT8 by NNCF.

Quantization Parameters

Weight compression was performed using nncf.compress_weights with the following parameters:

• mode: INT8_ASYM
• group_size: -1

For more information on quantization, check the OpenVINO model optimization guide.

Compatibility

The provided OpenVINO™ IR model is compatible with:

• OpenVINO version 2026.0.0 and higher
• Optimum Intel 1.27.0 and higher

Running Model Inference with Optimum Intel

1. Install packages required for using Optimum Intel integration with the OpenVINO backend:

pip install "git+https://github.com/huggingface/optimum-intel.git" "torch==2.8" --extra-index-url https://download.pytorch.org/whl/cpu

2. Run model inference:

import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer

from optimum.intel import OVModelForFeatureExtraction


model_id = "OpenVINO/Qwen3-Embedding-0.6B-int8-ov"

model = OVModelForFeatureExtraction.from_pretrained(model_id, export=False)


def last_token_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
    left_padding = attention_mask[:, -1].sum() == attention_mask.shape[0]
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f"Instruct: {task_description}\nQuery:{query}"


# Each query must come with a one-sentence instruction that describes the task
task = "Given a web search query, retrieve relevant passages that answer the query"

queries = [get_detailed_instruct(task, "What is the capital of China?"), get_detailed_instruct(task, "Explain gravity")]
# No need to add instruction for retrieval documents
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")

max_length = 8192

# Tokenize the input texts
batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=max_length,
    return_tensors="pt",
)
batch_dict.to(model.device)
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = embeddings[:2] @ embeddings[2:].T
print(scores.tolist())

For more examples and possible optimizations, refer to the Inference with Optimum Intel.

Running Model with OpenAI client and OpenVINO Model Server

1a. Deploy model on Windows using binary package:

mkdir C:\models
ovms.exe --rest_port 8000 --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov --model_repository_path C:\models

1b. Deploy model in a Docker container:

mkdir -p ${HOME}/models
export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models openvino/model_server:latest-gpu \
--rest_port 8000 --model_repository_path /models --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov
2. Install the client library:

pip install openai "numpy<2"

3. Run the client:

from openai import OpenAI import numpy as np

client = OpenAI( base_url="http://localhost:8000/v1", api_key="unused" ) model = "OpenVINO/Qwen3-Embedding-0.6B-int8-ov" embedding_responses = client.embeddings.create( input=[ "That is a happy person", "That is a very happy person" ], model=model, ) embedding_from_string1 = np.array(embedding_responses.data[0].embedding) embedding_from_string2 = np.array(embedding_responses.data[1].embedding) cos_sim = np.dot(embedding_from_string1, embedding_from_string2)/(np.linalg.norm(embedding_from_string1)*np.linalg.norm(embedding_from_string2)) print("Similarity score as cos_sim", cos_sim)

## Limitations

Check the original [model card](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) for limitations.

## Legal information

The original model is distributed under [Apache License Version 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) license. More details can be found in [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B).

## Disclaimer

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.