pplx-embed-v2-late-9b ONNX, 4/8-bit quantized

Quantized ONNX versions of perplexity-ai/pplx-embed-v2-late-9b (text tower) for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (MIT) belong to Perplexity. It is a late-interaction (ColBERT-style) retriever: one L2-normalized 128-d vector per token, scored with MaxSim. All 8 variants we built are here, including those below our quality gate, with their measurements.

Recommended: e8-w4-gptq-a4-ps, 5,425 MB instead of 31,759 MB, 6,518 MB RAM; nDCG@10 vs FP32 -0.14 [-0.87, +0.60] points on FreshRet Small, -0.04 [-0.70, +0.68] on MIRACL.

Use it with the 0.6B: index with the 9B, query with the 0.6B

pplx-embed-v2-late-0.6b and -9b share one embedding space and one tokenizer, so a corpus indexed with the 9B can be searched with 0.6B queries. This is the index side. Embed your documents once with this 9B model, then embed each query with the 0.6B (nynefold/pplx-embed-v2-late-0.6b-ONNX):

The 0.6B-query / 9B-index pair matches the 9B on both sides (-0.03 [-1.36, +1.26] points, paired) and beats the 0.6B on both sides (+1.28 [+0.01, +2.53]). The 9B runs once per document, offline; per query, only the query goes through the 0.6B. The index has one vector per document token whichever model built it, so it costs the same storage. Query-side quantization barely matters: every 0.6B variant, down to e4-w4-gptq-a4-pc (255 MB), keeps the 9B's quality as the query encoder on FreshRet Small, and the 9B index can be stored at INT8 (46 MB for 3,000 short documents) without measurable loss. For comparison, the best embedder + reranker pipeline we published (pplx-embed-v1 top 20 reranked by zerank-1-small) scores 0.870, but reads 20 query-document pairs through a 1.7B model per query. Adding that reranker on top of these models gives 0.880 (+4.42 [+2.39, +6.51] over the 0.6B / 9B pair alone, +0.93 [+0.27, +1.66] over pplx-embed-v1 + zerank), but then a 0.6B index does as well as a 9B one (0.879): the 9B index pays off when there is no reranker. Details, MIRACL and the full comparison: pplx-embed-v2-late findings.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/pplx-embed-v2-late-9b-ONNX inference.py pair_example.py --local-dir .

from inference import load, maxsim

model = load("nynefold/pplx-embed-v2-late-9b-ONNX")  # recommended variant; load("nynefold/pplx-embed-v2-late-9b-ONNX", variant="...") for another
docs = model.embed_documents(["Hold the reset button for ten seconds.", "The router has four LAN ports."])
query = model.embed_queries(["How do I reset the router?"])[0]
print(maxsim(query, docs))  # one score per document, higher is more relevant

pair_example.py does the same with the 9B for the documents and the 0.6B for the query (it downloads both picks, about 5,863 MB). embed_documents returns one [tokens, 128] float32 array per document with punctuation tokens already removed: that is the index to store (FP16 or per-vector INT8 storage cost nothing measurable on FreshRet Small). inference.py needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder). To run the graphs another way:

• Inputs input_ids and attention_mask, int64 [batch, seq], right-padded. Output token_embeddings float32 [batch, seq, 128], L2-normalized per token.
• Text: [Q] + query (at most 1,024 tokens) or [D] + document (at most 4,096 tokens), tokenizer default special tokens; [Q] / [D] are single added tokens. Use the variant's own tokenizer.json if it has one.
• Score: drop padding; for documents also drop the punctuation tokens listed in config.json (document_skiplist); then sum over query tokens of the best dot product with any document token (MaxSim).
• config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from (plus tokenizer.json for pruned vocabularies). Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

• Recommended variant chosen on FreshRet Small quality (smallest variant whose whole 95% interval stays above -1.0 point); MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries. Stage A: mean token cosine >= 0.99 and its 1st percentile >= 0.97.
• -ps variants: token table pruned to tokens made of Latin, Arabic or CJK characters, digits, punctuation, symbols and partial UTF-8 bytes; they carry their own tokenizer.json. Text in those scripts is almost always tokenized as before (99.5% of FreshRet Small texts identical); text in other scripts (Cyrillic, Greek, Devanagari, Hangul, kana, ...) is split into smaller pieces than the model was trained on, untested. For such text use e8-w4-gptq-a4.
• The FP32 reference outputs of the 9B come from PyTorch (its FP32 ONNX, 31.8 GB, does not fit our machine's RAM next to ONNX Runtime); the quantized variants were measured in ONNX Runtime.
• RAM: peak process memory while embedding one 256-token passage, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
• a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights (references: they give the same vectors within 1e-4 here).
• Tested with ONNX Runtime 1.31. Only the text tower is exported; images are not supported. Variant names: nynefold/retrieval-quant, plus ps / pc above.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN; mixed 4/8-bit from a per-tensor sensitivity scan). The Gated DeltaNet layers' chunked delta rule is exported with an ONNX Scan over 64-token chunks. Method, gates and all results: pplx-embed-v2-late findings in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small. Both repos are in the collection pplx-embed-v2-late ONNX: index with 9B, query with 0.6B.

Licence

MIT, as the base model: see LICENSE. Please cite and credit the original model, perplexity-ai/pplx-embed-v2-late-9b:

@article{krimmel2026pplxembed,
  title   = {{pplx-embed-v2-late} Technical Report},
  author  = {Krimmel, Markus and Milliken, Louis and Glazkova, Ekaterina and Schall, Maximilian and Chaffin, Antoine and Eslami, Sedigheh and Golde, Jonas and Bykov, Denis and Beamer, Claire and Yang, Jeremy and Wang, Bo},
  journal = {arXiv preprint arXiv:2610.29523},
  year    = {2026},
  doi     = {10.48550/arXiv.2610.29523},
  url     = {https://arxiv.org/abs/2610.29523}
}