zerank-2-reranker-vllm-fp8

FP8 version of polaria-tech/zerank-2-reranker-vllm, ZeroEntropy's zerank-2 reranker converted for vllm serve. Half the size (4.2 GB instead of 7.6 GB), same usage.

Everything in the bf16 model card applies here, in particular the three things that silently break the scores (--chat-template is mandatory, use the provided score_template.jinja, truncate with max_tokens_per_doc). This card only covers what differs.

All credit for the model goes to ZeroEntropy. Not affiliated with ZeroEntropy.

Quick start

TEMPLATE=$(hf download polaria-tech/zerank-2-reranker-vllm-fp8 score_template.jinja --quiet)

vllm serve polaria-tech/zerank-2-reranker-vllm-fp8 \
  --runner pooling \
  --max-model-len 32768 \
  --chat-template "$TEMPLATE"

--chat-template becomes optional from the first vLLM release after 0.32, which reads the template declared by the Sentence Transformers files (vllm-project/vllm#60877). Keeping it works on every version.

No --quantization flag: vLLM reads the FP8 configuration from the checkpoint. Requests are the same as for the bf16 model (/rerank, /score, 0-1 score sigmoid(logit / 5)).

FP8 matrix multiplications need a GPU with native FP8: Ada (e.g. L4, L40S, RTX 40xx), Hopper (H100, H200) or newer.

Like the bf16 model, the checkpoint carries Sentence Transformers files that declare the score template. vLLM ≤ 0.32 ignores them (--chat-template stays mandatory); since vllm-project/vllm#60877 (merged 2026-10-18, after 0.32.0), vLLM uses them to pick the template automatically. Loading this FP8 checkpoint with Sentence Transformers is untested: use the bf16 model there.

Quantization

Made with llm-compressor 0.14, scheme FP8_DYNAMIC, no calibration data:

• Weights of the 252 Linear layers of the decoder: FP8 E4M3, one scale per output channel, round-to-nearest.
• Activations: FP8, one scale per token, computed at runtime. Nothing is calibrated on a dataset, so there is no bias towards a language or a document length.
• Kept in full precision: the score head (score.weight, [1, 2560]), in fp32, bit-identical to the bf16 model, since it produces the final score; embeddings and norms stay in bf16.

The quantization script is quantize_fp8.py in github.com/polaria-tech/zerank-2-reranker-vllm.

Quality

The left panel is the bf16 model; FP8 is about 1.2× faster again (see Speed below).

Reranking quality on the MTEB reranking tasks (fixed candidate lists, so only the reranker is measured), NDCG@10, vLLM 0.31.0 on an H100 PCIe, documents truncated to 4,000 tokens:

FP8 costs 0.001 to 0.0025 NDCG@10 (0.1 to 0.25 points) on the larger tasks. SyntecReranking only has 100 queries: its +0.0037 is within noise. bf16 scores are reproducible to 5 decimals across runs, so the small losses are real, not run-to-run noise.

Speed, same job and settings, bf16 then FP8 (time to score each task):

Measured on a time-sliced H100 PCIe shared with other models, with a small KV cache (1.25 GiB): a dedicated GPU with larger batches should gain more from FP8, but this was not measured.

Agreement with the bf16 model, on the 40 pairs of the bf16 parity set (9 queries, English, French, Japanese, 19 → 6,051 tokens), vLLM nightly (October 2026), RTX 4080 SUPER:

FP8 scores are not bit-identical to the bf16 ones and should not be compared to them with the bf16 noise tolerance. Judge the FP8 model on rankings and retrieval metrics.

Limitations

• Validated with vLLM 0.31.0 (H100 PCIe) and a vLLM nightly build (RTX 4080 SUPER). FP8 kernels differ between GPUs, so scores can vary slightly from one GPU to another.
• The long-document behaviour and the context limits described in the bf16 card apply here too.

License and attribution

Apache 2.0, like the original. Model by ZeroEntropy: zeroentropy/zerank-2-reranker, technical report arXiv:2509.12541. Conversion, quantization and evaluation by Polaria.