harrier-oss-v1-270m ONNX INT8

An 8-bit build of microsoft/harrier-oss-v1-270m that Text Embeddings Inference (TEI) loads on CPU. It's quantized from the full-precision ONNX export in onnx-community/harrier-oss-v1-270m-ONNX, and Parabol uses it in production.

TEI's ONNX backend only reads onnx/model.onnx and onnx/model.onnx_data, so the other quantized files we tried either couldn't be selected or used an operator TEI's ONNX Runtime can't run (GatherBlockQuantized). This build quantizes MatMul and Gather (including the 262,144 × 640 token embedding) to per-channel symmetric UINT8 with standard operators. The weights are 302 MB, against 1,106 MB at full precision.

Usage with TEI

docker run -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.4 \
  --model-id Parabol/harrier-oss-v1-270m-ONNX-INT8 --pooling last-token

Use last-token pooling. As with the original model, put an instruction before queries and leave documents as they are, for example:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: <your query>

How it compares to full precision

Measured on TEI 1.9.4 (x86-64, 3 CPUs) against the full-precision ONNX export:

On Parabol's internal search evaluation sets, ranking quality was equal to full precision within 0.01 nDCG@10. Results aren't identical: similarity scores move by about 0.01 on average, so near-ties can swap places and pages close to a fixed similarity cutoff can cross it.

Reproducing it

quantize.py rebuilds onnx/model.onnx and onnx/model.onnx_data byte for byte with onnxruntime 1.22.1 and onnx 1.23.2, starting from onnx-community/harrier-oss-v1-270m-ONNX at revision d59c919d0159aea2c19ed7d04288fcdd048d0f9c. The tokenizer and config files are copied from that revision unchanged.

License

MIT, the same as microsoft/harrier-oss-v1-270m. All credit for the model goes to its authors at Microsoft, and for the ONNX export to onnx-community.