zerank-2-reranker-vllm

ZeroEntropy's zerank-2 reranker, ready for vllm serve. Same weights, same scores, about twice as fast as the official path.

Serving zerank-2 on vLLM is deceptively easy to get wrong: the server starts, every request returns 200, and the scores are quietly off. We converted the model, measured it against the official implementation, benchmarked it on an H100 and wrote down every trap we hit, so you can just serve it.

• The official 0-1 score, sigmoid(logit / 5), straight from /rerank and /score. No --hf-overrides, no client-side prompt formatting.
• Same scores as CrossEncoder("zeroentropy/zerank-2-reranker").predict(), within bf16 noise (see Parity).
• Faster: on an H100, a rerank of 100 documents takes 0.62 s instead of 1.19 s, and throughput goes from 84 to 184 pairs/s (see Benchmark).

[!TIP] Also available in FP8: polaria-tech/zerank-2-reranker-vllm-fp8. Same results (NDCG@10 within ±0.004 on 4 MTEB reranking tasks), ~1.2x faster on H100 and half the size (4.2 GB). Needs a GPU with native FP8 (Ada, Hopper or newer).

This is a format conversion of zeroentropy/zerank-2-reranker (formerly zeroentropy/zerank-2) into Qwen3ForSequenceClassification, not a new model: all credit for the model goes to ZeroEntropy. Converted from revision 5eae30d5, after ZeroEntropy relicensed the model under Apache 2.0 on 2026-07-24. Not affiliated with ZeroEntropy.

Validity. Validated in October 2026 with vLLM 0.29.0 and 0.31.0, sentence-transformers 6.1.0, on H100. The --chat-template requirement and the truncation caveats below are vLLM behaviours of those versions; re-run the parity script before relying on a newer vLLM. Loading with Sentence Transformers checked with sentence-transformers 5.7.0.

Quick start

pip install "vllm>=0.29,<=0.31" huggingface_hub   # validated range, see Validity

TEMPLATE=$(hf download polaria-tech/zerank-2-reranker-vllm score_template.jinja --quiet)

vllm serve polaria-tech/zerank-2-reranker-vllm \
  --runner pooling \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --chat-template "$TEMPLATE"

--chat-template becomes optional from the first vLLM release after 0.32, which reads the template declared by the Sentence Transformers files (vllm-project/vllm#60877). Keeping it works on every version.

--max-model-len 32768 matches zerank-2's advertised context. Without it, vLLM uses the Qwen3 config's 40,960 positions; scores stay faithful to the original model up to 38k tokens (tested), but ZeroEntropy only advertises 32k.

import requests

resp = requests.post("http://localhost:8000/rerank", json={
    "model": "polaria-tech/zerank-2-reranker-vllm",
    "query": "What is 2+2?",
    "documents": ["4", "The answer is definitely 1 million"],
    "max_tokens_per_doc": 8000,  # optional, the only safe way to truncate (see below)
}).json()

for r in resp["results"]:
    print(r["index"], round(r["relevance_score"], 4))
# 0 ≈ 0.75   (raw logit ≈ 5.4)
# 1 ≈ 0.29   (raw logit ≈ -4.5)

/score works the same way (text_1 = query, text_2 = documents). To get the raw logit that CrossEncoder.predict() returns: logit = 5 * log(s / (1 - s)).

Custom instructions

zerank-2 is an instruction-following reranker (see ZeroEntropy's prompting guide). Two equivalent ways to pass an instruction:

# 1. vLLM's `instruction` field (also works on /score and the Cohere-style /v2/rerank)
resp = requests.post("http://localhost:8000/rerank", json={
    "model": "polaria-tech/zerank-2-reranker-vllm",
    "query": "VAT limitation period",
    "instruction": "We are a French tax law firm. Prefer statutes and recent French case law "
                   "over blog posts and commentary.",
    "documents": documents,
}).json()

# 2. Inside the query string, in any readable format, as in ZeroEntropy's guide
query = (
    "<query>VAT limitation period</query>\n"
    "<instruction>We are a French tax law firm. Prefer statutes and recent French case law "
    "over blog posts and commentary.</instruction>"
)

The template turns the instruction field into exactly the second form. Without an instruction, the prompt is the official zerank-2 prompt, unchanged: no default instruction is added, since zerank-2 is meant to work from the query alone. (Templates that ignore the instruction variable, including zerank-2's original one, silently drop that field.)

Long instructions are cheap: with prefix caching (on by default), the query is computed once per rerank call and shared by all its documents. In the benchmark below, a ~300-token query costs 0.69 s per 100 documents with the cache, 1.11 s without.

Sentence Transformers

The checkpoint is also a Sentence Transformers CrossEncoder (one sequence-classification module, with the score template saved as its chat template), so it loads directly with sentence-transformers ≥ 5.4:

import torch
from sentence_transformers import CrossEncoder

model = CrossEncoder("polaria-tech/zerank-2-reranker-vllm", model_kwargs={"dtype": torch.bfloat16})
scores = model.predict([
    ("What is 2+2?", "4"),
    ("What is 2+2?", "The answer is definitely 1 million"),
])  # 0-1 scores, sigmoid(logit / 5), like vLLM

Unlike CrossEncoder("zeroentropy/zerank-2-reranker"), predict() returns the 0-1 score, not the raw logit. On the 40 pairs of the parity set (sentence-transformers 5.7.0, both in bf16): prompt token ids identical to vLLM's on 40/40 pairs, max deviation 0.30 logit / 0.008 score.

⚠️ Three things that silently break the scores

None of these raise an error: the server answers, the scores are just wrong.

1. --chat-template is mandatory. vLLM (≤ 0.32) ignores the checkpoint's template for scoring and falls back to concatenating query and document. Without the flag: mean error of 11.4 logit and many ranking inversions. Since vllm-project/vllm#60877 (merged 2026-10-18, after 0.32.0), vLLM uses the template declared by this checkpoint's Sentence Transformers files and logs Scoring with the 'default' chat template declared by the Sentence Transformers config at startup. Until you run a release that includes it, pass the flag; passing it remains harmless.
2. Use the provided score_template.jinja, not the original zerank-2 chat template. vLLM renders score templates without add_generation_prompt, so the original template drops the final <|im_start|>assistant\n and the score is read at the wrong position (mean error 9.9 logit).
3. Truncate with max_tokens_per_doc, never with truncate_prompt_tokens. The latter cuts the end of the prompt (the template suffix, or the query with truncation_side=left), producing wrong scores. Keep query tokens + document tokens + 13 < --max-model-len.

Also note a vLLM scheduler bug (0.29 and 0.31): a prompt of exactly --max-model-len tokens never completes (the request hangs without error). One token less works. truncate_prompt_tokens=-1 always hits it. Fixed upstream by vllm-project/vllm#48039 (merged 2026-10-07), not yet in a release as of 0.31.1rc0.

Recommended settings (H100)

vllm serve polaria-tech/zerank-2-reranker-vllm \
  --runner pooling \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --chat-template "$TEMPLATE" \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.3

• Keep prefix caching on (the default). The query sits at the head of every prompt of a rerank call, so it is computed once for all documents: −35% latency with ~300-token queries.
• --max-num-batched-tokens 65536: +5% throughput over the default 8192.
• --gpu-memory-utilization 0.3 (~24 GiB) performs the same as 0.92. Reranking is compute-bound, not memory-bound: weights are 7.8 GiB and the KV cache is short-lived. With 0.3, the KV cache holds ~77k tokens, enough for a full 32k (or 40k) prompt; below ~0.25 a 40k prompt no longer fits.

Benchmark

H100 80GB SXM, vLLM 0.31.0, bf16. One rerank call = 1 query + 100 documents of 300–500 tokens. "Short" queries ≈ 15 tokens, "long" queries ≈ 300 tokens. Every call has a unique query (no cross-call cache hits). 32+ measured calls per cell after warm-up.

sentence-transformers = the official CrossEncoder.predict() path, bf16, best batch size (32 of 16–128). It processes one call at a time, so its saturation throughput is its single-call throughput.

Under load (recommended settings, short queries)

The GPU saturates from ~8 concurrent calls, at ~79k prompt tokens/s. Beyond that, extra calls queue and latency grows linearly: size client concurrency to your latency budget. Cost scales with total tokens, roughly n_docs × (query + doc + 13).

Parity with the official model

Reference: CrossEncoder("zeroentropy/zerank-2-reranker").predict() (sentence-transformers 6.1.0), which returns the raw Yes logit; its 0-1 score is (predict() / 5).sigmoid(). On the vLLM side, the raw logit is recovered as 5 * log(s / (1 - s)), and both logits and 0-1 scores are compared.

Ground truth: the same official model in fp32. The official bf16 path is itself not exact, so its deviation from fp32 is used as the noise floor, and vLLM must stay within 1.5x of it.

• Main set: 40 pairs, 9 queries, multilingual, 19 → 6,051 tokens, mixed lengths in every batch. Long set: 7 pairs, 48 → 38,203 tokens.
• Prompt token ids identical to the official path on 47/47 pairs.
• On these 9 queries: top-1 identical to fp32 everywhere, and no ranking inversion between documents further apart than 2x the noise floor. The sample is sized to measure bf16 noise, not to certify ranking quality on your data.
• No extra drift with length.

vLLM ends up closer to fp32 than the official bf16 path, because the score head runs in fp32.

How it was converted

zerank-2 is a Qwen3ForCausalLM (Qwen3-4B) whose score is the logit of the token Yes (id 9454) at the last position, with no negative token; the 0-1 score is sigmoid(logit / 5).

• score.weight [1, 2560] = model.embed_tokens.weight[9454] / 5, stored in fp32 (zerank-2 has tied embeddings, so there is no lm_head.weight). vLLM applies a sigmoid for num_labels=1 and runs pooling heads in fp32, so it outputs sigmoid(logit / 5) with no extra rounding.
• The backbone shards are copied unchanged.
• config.json: Qwen3ForSequenceClassification, num_labels: 1, tie_word_embeddings: false, use_sep_token: false. None of the keys (method, classifier_from_token, is_original_qwen3_reranker) that make vLLM re-derive the head from lm_head at load time.
• Sentence Transformers files: a single-module CrossEncoder (transformer_task: sequence-classification, message modality, Sigmoid activation) scored with the saved template. The original files (causal LM + LogitScore, Identity activation) are not reused. vLLM ≤ 0.32 ignores these files for scoring (same prompt token ids and scores with and without them); later versions read only the chat template from them.
• The Qwen3-Reranker recipe (no/yes pair, from_2_way_softmax) does not apply to zerank-2 and gives compressed, wrong scores.

The prompt is:

<|im_start|>system
{query}<|im_end|>
<|im_start|>user
{document}<|im_end|>
<|im_start|>assistant

Conversion script, reference and parity scripts, truncation tests and benchmark: github.com/polaria-tech/zerank-2-reranker-vllm.

Long documents

This is the original model's behaviour (identical in fp32), not a conversion artifact: relevance gets diluted in long documents, most when the relevant passage sits in the middle.

Ordering stays correct, but chunking long documents is advisable.

Limitations

• Tested with vLLM 0.29.0 and 0.31.0 only, in bf16. fp16 untested.
• Advertised context 32,768 tokens. Parity tested up to 38k tokens per pair, which shows the conversion is faithful there, not that the model is reliable beyond 32k.
• Built for vLLM and Sentence Transformers. Loading it with plain transformers (AutoModelForSequenceClassification, without the template) is untested.

License and attribution

Apache 2.0, like the original. Model by ZeroEntropy: zeroentropy/zerank-2-reranker (formerly zeroentropy/zerank-2; relicensed under Apache 2.0 on 2026-07-24, converted from revision 5eae30d5ee3c6b2df2ef6d723bde45172d761c4c), technical report arXiv:2509.12541. Conversion and evaluation by Polaria.