Qwen3-Embedding-4B — ONNX int8

An int8 ONNX export of Qwen/Qwen3-Embedding-4B for CPU inference with ONNX Runtime: 4.2 GB instead of ~16 GB in fp32, with embeddings at cosine ≥ 0.999 to the fp32 model.

Inputs: input_ids, attention_mask (int64, [batch, sequence]). Output: last_hidden_state (float32, [batch, sequence, 2560]). Pool the last non-padding token, then L2-normalize, as for the original model.

Quantization

• Linear layers (all 252 projections): 8-bit symmetric weights, one scale per block of 32 input channels (com.microsoft::MatMulNBits, bits=8, block_size=32, accuracy_level=4: ONNX Runtime also quantizes activations per block, which keeps Qwen3's activation outliers local).
• Token embeddings: int8, one scale per row.
• Norms and rotary frequencies: fp32.

Per-tensor dynamic int8 (onnxruntime.quantization.quantize_dynamic) degrades this model noticeably; block-wise weights avoid that.

Quality

Compared with the fp32 PyTorch model on 12 texts (EN / FR / DE / JA, code, short and 160-token inputs):

Query × document scores of the original model card example:

Padded batches give the same embeddings as rows run one at a time (cosine 1.0).

Usage

import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

path = snapshot_download("mariusbrt/Qwen3-Embedding-4B-ONNX-int8")
tok = AutoTokenizer.from_pretrained(path)
sess = ort.InferenceSession(f"{path}/model_int8.onnx", providers=["CPUExecutionProvider"])


def embed(texts):
    enc = tok(texts, padding=True, truncation=True, max_length=8192, return_tensors="np")
    mask = enc["attention_mask"].astype(np.int64)
    hidden = sess.run(None, {"input_ids": enc["input_ids"].astype(np.int64), "attention_mask": mask})[0]
    last = mask.shape[1] - 1 - np.argmax(mask[:, ::-1], axis=1)  # last real token, either padding side
    emb = hidden[np.arange(len(texts)), last]
    return emb / np.linalg.norm(emb, axis=1, keepdims=True)


task = "Given a web search query, retrieve relevant passages that answer the query"
queries = [f"Instruct: {task}\nQuery:{q}" for q in ["What is the capital of China?", "Explain gravity"]]
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]
print(embed(queries) @ embed(documents).T)
# [[0.7486 0.1121]
#  [0.0317 0.6242]]

As with the original model, queries take a one-sentence instruction and documents none.

Serving with rs-infer

rs-infer (docs) is a Rust inference server on ONNX Runtime. It serves this repository straight from the Hub with an OpenAI-compatible API, applies the last-token pooling from 1_Pooling/config.json and returns L2-normalized vectors.

# config.yaml
models:
  - name: qwen3-embedding-4b
    kind: embedding
    hf: mariusbrt/Qwen3-Embedding-4B-ONNX-int8
    file: model_int8.onnx
    max_len: 512   # truncation; attention memory grows with the square of the length
    replicas: 1    # each replica holds its own copy of the weights (~4.7 GB of RAM)
    eps: [cpu]

make cpu
./target/release/rsinfer-server --config config.yaml   # first start downloads 4.2 GB

curl http://localhost:8080/v1/embeddings -H 'Content-Type: application/json' -d '{
  "model": "qwen3-embedding-4b",
  "input": [
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:What is the capital of China?",
    "The capital of China is Beijing."
  ]
}'

The server does not add the instruction: prefix queries yourself with Instruct: {task}\nQuery:{query} and send documents as they are.

Performance

CPU only (Intel i7-13850HX, ONNX Runtime 1.29, Python): ~260 ms for a short query, ~80 tokens/s on large batches. At large batch sizes the int8 kernels run at roughly the speed of fp32 on this CPU; the gain is memory (4.2 GB instead of ~16 GB) and small-batch latency.

Limitations

• MatMulNBits is an ONNX Runtime contrib operator: other ONNX runtimes cannot run this graph.
• Tested on the CPU execution provider only; GPU providers are untested.
• Quality was checked on inputs up to 161 tokens.

Credits

Original model: Qwen/Qwen3-Embedding-4B by the Qwen team, Apache-2.0. This repository only changes the format and the weight precision.

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}