pplx-embed-v1-0.6b ONNX, 4/8-bit quantized

Quantized ONNX versions of perplexity-ai/pplx-embed-v1-0.6b for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (MIT) belong to its original authors. All 26 variants we built are here, including those below our quality gate, with their measurements.

Recommended: mix02-gptq-a4-pc, 401 MB instead of 2,387 MB, 571 MB RAM instead of 1,892 MB; nDCG@10 vs FP32 -0.08 [-0.91, +0.74] points on FreshRet Small, -0.38 [-1.38, +0.55] on MIRACL. Unpruned alternative: mix02-gptq-a4, 470 MB (see the -pc note under Variants).

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/pplx-embed-v1-0.6b-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/pplx-embed-v1-0.6b-ONNX")  # recommended variant; load("nynefold/pplx-embed-v1-0.6b-ONNX", variant="...") for another
q = model.embed_queries(["How do I reset the router?"])
d = model.embed_documents(["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(q @ d.T)  # cosine similarities (vectors are L2-normalized)

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

• Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
• No query or document prefix. At most 512 tokens.
• Mean pooling is inside the graph; output embedding [batch, dim].
• Apply round(127 * tanh(x)) to get the base model's native int8 embedding, then compare by cosine (that is what was evaluated).
• config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

• Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
• "Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
• RAM: peak process memory while embedding one 256-token passage, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
• a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
• Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.
• -pc variants: vocabulary pruned to the tokens seen in the MIRACL corpus (six languages; -69 MB); they carry their own tokenizer.json. Text with other tokens is split differently: on FreshRet Small (not used for the pruning) this costs -0.29 points [-0.78, +0.22] overall against the unpruned mix02-gptq-a4, and -1.00 [-1.95, -0.27] on Arabic documents. Use mix02-gptq-a4 (470 MB) where that matters.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.

Licence

MIT, as the base model: see LICENSE. Please cite and credit the original model, perplexity-ai/pplx-embed-v1-0.6b.