bge-reranker-v2-m3, SmoothQuant int8 ONNX

An int8 ONNX build of BAAI/bge-reranker-v2-m3, the multilingual cross-encoder. It is a drop-in replacement for onnx/model_int8.onnx of onnx-community/bge-reranker-v2-m3-ONNX: it has the same inputs, outputs and tokenizer, nearly the same size (+0.04%), and runs on any CPU that runs the upstream file. Its scores are closer to the fp32 model's. PaminLabs uses it as the reranker of Påmin Memory.

Why

The upstream int8 file was made with onnxruntime's dynamic quantization:

• weights: int8 per output channel;
• activations: uint8 with one scale per tensor, computed at run time.

In this XLM-RoBERTa-large model, the inputs to the q/k/v projections and to the first feed-forward layer are LayerNorm outputs with a few outlier channels. Typically 3 to 5 of 1,024 channels exceed a quarter of the tensor's maximum. Those channels set the single activation scale, so every other channel is quantized coarsely.

SmoothQuant moves that range from the activations into the weights. For each such input it divides channel j by a factor s_j and multiplies the matching weight row by the same s_j. The product is unchanged before quantization, but both sides become easier to quantize. Here s_j = absmax_j^0.75 / wmax_j^0.25, where:

• absmax_j is the channel's activation maximum over calibration pairs;
• wmax_j is the largest weight in that row.

Concretely:

• 96 weights re-quantized. q/k/v (sharing one factor) and the first feed-forward layer, in all 24 layers, are re-quantized from the fp32 weights after scaling. The arithmetic is the upstream recipe's: symmetric per output channel, in [-64, 64], round-half-even. The 7-bit range is what keeps the file safe on AVX2-only CPUs.
• 48 Mul nodes added. Each multiplies one smoothed input by 1/s just before its activation quantizer. These are the only graph change: everything else is upstream's, including the other 50 weights, byte for byte.

Results

Measured against the upstream model_int8.onnx at 6f5ff652. Both files ran in the same onnxruntime 1.28 CPU build, at batch 1, on the same pairs. The calibration pairs were held out of every evaluation below.

Agreement with the fp32 model. Both int8 files and the fp32 model scored the same candidate lists. The table gives Kendall's τ-b of each int8 file's ranking against fp32's (mean over lists), and the median |logit difference|.

Ranking quality. nDCG@10 after reranking retrieval shortlists, each query's list as one unit. The difference is this file minus upstream; p comes from a paired sign-flip test over units.

Speed, memory and size. CPU only: a 4-core Xeon with AVX-512 VNNI and no AMX, onnxruntime 1.28.

• Pairs. The same 120 real pairs (32–256 tokens, median 106), one pair per pass.
• Ratio. Total time this / upstream, with a 95% CI, from 8 rotated repetitions.
• Graph. Both files ran as Påmin Memory runs them: onnxruntime's full graph optimization, with the last layer computed for the first token only, which is all a classification head reads.

In short, it agrees better with fp32 and gains about a quarter of a point of cross-lingual nDCG@10. It costs about 1.5% of single-stream time and no measurable time under concurrency. GPU and NPU users are unaffected: onnx/model_fp16.onnx is upstream's file, unchanged. ARM, AVX2-only x86 and other execution providers were not benchmarked (N/A), though the file uses only the operators the upstream file does, plus Mul.

In a whole search engine. The numbers above score the reranker alone. Påmin Memory also measured it as its reranker, on the CPU, through the full search path: retrieval, fusion and this reranker over the shortlist. The run covered all 1,190 XQuAD-R questions against 13,014 sentences in eleven languages. Each file ran in a fresh index.

In the same product, search latency, peak memory and disk showed no significant change. Over the searches that run the models (each query's first time, so no cached score is reused), the p50 ratio was 0.969 [0.919, 1.104] and the p95 ratio 1.010 [0.964, 1.185]. Those runs had 4 rotated runs per file, 120 queries each, on the 4-core Xeon above.

Usage

import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=512)
session = ort.InferenceSession("onnx/model_int8.onnx", providers=["CPUExecutionProvider"])

def score(query, passage):
    e = tok.encode(query, passage)
    ids = np.array([e.ids], np.int64)
    logits = session.run(None, {"input_ids": ids, "attention_mask": np.ones_like(ids)})[0]
    return float(logits[0, 0])  # higher = more relevant; sigmoid for a 0-1 score

Any code that loads the upstream onnx/model_int8.onnx can load this file instead.

Files

How it was built

1. Calibration. recipe/calibrate.py ran the upstream fp32 model over 520 query–passage pairs at batch 1, up to 256 tokens each. It recorded each channel's maximum |activation| at the 48 smoothed inputs. The pairs came from reranking shortlists of XQuAD-R, MuSiQue and LoCoMo (200, 120 and 200 pairs), spread over lengths. They were drawn only from queries and conversations held out of every evaluation above.
2. Export. recipe/export.py --alpha 0.75 applied the factors to the upstream fp32 weights and re-quantized them into the upstream int8 graph.
3. Checks.
   • calibrate.py reproduces calibration_absmax.npz exactly from the same pairs.
   • export.py reproduces onnx/model_int8.onnx byte for byte (recipe/sync.sh), with numpy 2.5 and onnx 1.23.
   • The file scores pairs identically to the graph that was evaluated.

Alpha 0.75 was chosen over 0.5, which lost ranking quality on two sets. The attention-output and second feed-forward inputs were left unsmoothed: their large values are spread over hundreds to thousands of channels, not concentrated in a few, so a per-channel factor has little to move.

Keeping up with upstream

This build is pinned to onnx-community/bge-reranker-v2-m3-ONNX at 6f5ff65298512715a1e669753bc754d2bc8f367b; MANIFEST.json lists the hash of every upstream file used. When upstream publishes a new revision:

1. Run recipe/sync.sh <new revision>. It downloads that revision's int8 and fp32 files, rebuilds with the shipped calibration maxima, and compares the result with the published file.
2. Identical. The model is unchanged, for example after a README-only or tokenizer-config-only commit. Copy any changed tokenizer or config files and record the new revision in MANIFEST.json and CHANGELOG.md.
3. Different. Find out whether the fp32 weights changed: compare onnx/model.onnx_data's hash against MANIFEST.json.
   • The fp32 weights changed. The maxima no longer describe the model. Re-run recipe/calibrate.py on held-out pairs, then export.py.
   • Only the int8 graph changed. For example, a different quantizer version. Check that export.py still finds its 96 weights and 48 quantizers: it stops if it does not.
4. Evaluate before publishing. Compare the new file against the new upstream int8 on held-out ranking data: τ against fp32, and nDCG@10 with a paired test. Publish only if it is not significantly worse on any set.
5. Publish under a new commit. Update MANIFEST.json, checksums.sha256 and CHANGELOG.md, and the Results table if it was re-measured.

The BAAI weights themselves (BAAI/bge-reranker-v2-m3) are the root of the chain. A new BAAI release reaches here only through a new onnx-community export, and goes through the same steps.

License and attribution

Apache-2.0, as the base model. This is a modified version of BAAI/bge-reranker-v2-m3 (Apache-2.0), via onnx-community/bge-reranker-v2-m3-ONNX at 6f5ff652. PaminLabs modified only onnx/model_int8.onnx, as described above; see NOTICE and CHANGELOG.md.