zerank-1-small ONNX, 4/8-bit quantized

Quantized ONNX versions of zeroentropy/zerank-1-small-reranker for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (Apache-2.0) belong to its original authors. All 12 variants we built are here, including those below our quality gate, with their measurements.

Recommended: mix30-gptq-sq50-a4, 1,483 MB instead of 6,885 MB, 2,505 MB RAM instead of 6,445 MB; nDCG@10 vs FP32 -0.30 [-0.93, +0.33] points on FreshRet Small, -0.36 [-1.42, +0.54] on MIRACL.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/zerank-1-small-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/zerank-1-small-ONNX")  # recommended variant; load("nynefold/zerank-1-small-ONNX", variant="...") for another
scores = model.rerank("How do I reset the router?",
                      ["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(scores)  # relevance in [0, 1], one per document

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

• Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
• Input text: <|im_start|>system\n{query}<|im_end|>\n<|im_start|>user\n{document}<|im_end|>\n<|im_start|>assistant\n, query cut to 2,000 and document to 10,000 characters. Above 1024 tokens only the document's tokens are shortened, so the prompt always ends with the template's tail.
• Output logit [batch]: the logit of token Yes; score = sigmoid(logit / 5).
• config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

• Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
• "Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
• RAM: peak process memory while scoring 20 candidates of 170 tokens, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
• a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
• Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.

Licence

Apache-2.0, as the base model: see LICENSE. Please cite and credit the original model, zeroentropy/zerank-1-small-reranker.