zev-reranker 0.4.2

One model, two jobs:

• Decisions (predict): typed decision primitives — noul (yes/no), choice and score — over a text, JSON or conversation state.
• Reranking (rerank): a cross-encoder reranker with a null decision (null: true when no candidate looks relevant). It is a LoRA adapter on the same encoder plus a small head, switched on only for the duration of a rerank() call.

Both share one encoder (mmBERT-base, 22 layers, hidden size 768); the reranker adds about 15 MB.

Based on convaiinnovations/laya (multilingual, Apache-2.0): its decision weights are included unchanged, so predict gives the same answers. See NOTICE.

Usage

pip install "transformers>=5" torch

from transformers import AutoModel

model = AutoModel.from_pretrained("zafonair/zev-reranker", trust_remote_code=True)
# fp32 weights; on GPU with bf16 autocast if available, otherwise CPU. The tokenizer loads with the model.

Decisions

out = model.predict(
    "I want to cancel my subscription right now, this is the third time I ask!",
    {
        "intent": {"type": "choice", "instructions": "What does the customer want?",
                   "criteria": {"cancel": "end the subscription", "upgrade": "move to a bigger plan",
                                "billing": "a question about an invoice", "other": None}},
        "angry": {"type": "noul", "instructions": "Is the customer angry?"},
        "urgency": {"type": "score", "instructions": "How urgent is this?",
                    "criteria": ["not urgent", "somewhat urgent", "urgent", "extremely urgent"]},
    },
)
out["answers"]["intent"]["choice"]   # "cancel"
out["answers"]["angry"]["noul"]      # P(true)
out["answers"]["urgency"]["score"]   # expected level

• noul: {"type": "noul", "instructions": ..., "criteria": {"false": ..., "true": ...}, "labels": {"false": ..., "true": ...}} (criteria and labels optional); answer noul is P(true).
• choice: criteria is {label: description} or a list of labels; answer choice, probabilities.
• score: criteria is a list of level descriptions, index 0 first; answer score (expected level), probabilities.
• Each answer also has confidence, answer_confidence and action.act_probability. usage reports tokens and whether the state was truncated (max 1024 tokens per question).
• model.predict_batch(states, questions, batch_size=None) runs many states in shared forward passes.
• Not supported: hooks, long-document windowing, min_confidence gating, calibration files, language temperature overrides and fast/compile/ONNX backends.

Reranking

out = model.rerank(
    "What is the boiling point of water?",
    ["Water boils at 100 degrees Celsius at sea level.",
     {"id": "d7", "text": "Football is played with 11 players."}],
    top_k=10,
)
# {"results": [{"index": 0, "id": 0, "rank": 1, "score": 0.89, "probability": 0.71},
#              {"index": 1, "id": "d7", "rank": 2, "score": -7.16, "probability": 0.0002}],
#  "null": False, "null_probability": 0.29}

• Candidates can be strings or {"id", "text", ...} dicts; only text is sent to the model. Candidates are scored independently, so text inside one cannot affect another's score.
• Ties keep input order. top_k=None returns the full ranking, top_k=0 returns nothing.
• model.score(query, docs) returns raw scores (logits) in input order.
• Inputs are truncated to 512 tokens (query + document).

Decisions and reranking in one call

A rerank question can sit next to noul / choice / score questions in the same predict. For example, a search on a streaming platform: read the user's intent and rank the catalog in one call.

catalog = [
    {"id": "free-guy", "text": "Free Guy (2021). Comedy, action. A bank teller played by Ryan Reynolds "
                               "discovers he is a background character in a video game.", "metadata": {"rating": "PG-13"}},
    {"id": "deadpool", "text": "Deadpool (2016). Action comedy. Ryan Reynolds stars as a foul-mouthed "
                               "mercenary with healing powers.", "metadata": {"rating": "R"}},
    {"id": "the-proposal", "text": "The Proposal (2009). Romantic comedy with Sandra Bullock and Ryan Reynolds "
                                   "about a fake engagement.", "metadata": {"rating": "PG-13"}},
    {"id": "the-notebook", "text": "The Notebook (2004). Romantic drama about a young couple separated by class "
                                   "and war.", "metadata": {"rating": "PG-13"}},
    {"id": "planet-earth", "text": "Planet Earth II (2016). Nature documentary series narrated by David "
                                   "Attenborough.", "metadata": {"rating": "TV-G"}},
    {"id": "frozen", "text": "Frozen (2013). Animated musical for the whole family about two royal sisters "
                             "and a magical winter.", "metadata": {"rating": "PG"}},
]

search = "funny movies with ryan reynolds"
out = model.predict(search, {
    "genre": {"type": "choice",
              "instructions": "The text is a search typed into a streaming app. Which genre fits it best?",
              "criteria": {"comedy": "funny content", "romance": "love stories", "drama": "serious or sad stories",
                           "action": "fights, chases, explosions", "documentary": "real-world, nature or history",
                           "kids": "cartoons and animation for children", "any": "no genre is implied"}},
    "for_kids": {"type": "noul",
                 "instructions": "Is the search for children's content, like cartoons or family movies?"},
    "results": {"type": "rerank", "candidates": catalog, "top_k": 3},   # query: the search itself
})
a = out["answers"]
a["genre"]["choice"]                                  # "comedy"
a["for_kids"]["noul"] > 0.5                           # False
[r["id"] for r in a["results"]["results"]]            # ["the-proposal", "deadpool", "free-guy"]

Use the decisions to shape the results, e.g. filter on metadata (which is never sent to the model) when for_kids is true, or boost the detected genre.

• A rerank question takes candidates and optionally query (defaults to the state), top_k and null_threshold; its answer has the same shape as rerank() plus "type": "rerank".
• When the state is not search-like (e.g. a long customer message), pass an explicit query: the ranking tends to stay the same but scores drop, so null turns true (see the null threshold limitation below).
• Decisions are zero-shot. On very short search texts, word questions explicitly (describe each option in criteria) and check them on your own queries: in our quick tries a genre choice with descriptions was right on all 7 searches, while yes/no questions leaned towards false and a "searching by title / person / genre / mood" choice was unreliable.
• Answers come back in question order. Every question is validated before the model runs.
• It is one call, not one forward pass: decisions run with the adapter off, then reranking with it on, so the time is the sum of both. usage counts the decision questions' tokens only.
• In predict_batch, a rerank question without query ranks its candidates against each state.

Loading without the reranker

decisions_only = AutoModel.from_pretrained("zafonair/zev-reranker", trust_remote_code=True, load_reranker=False)

The adapter and head are not loaded; predict works as above, and rerank() or a rerank question raises.

Notes

• trust_remote_code=True runs the model code in this repo (modeling_zev.py, decision_zev.py, configuration_zev.py). To pin it, pass revision="<commit>".
• Pass dtype=... or device_map=... to control placement yourself.
• The adapter is switched on and off under a lock, so concurrent predict and rerank calls on one model instance run one at a time.
• Tested with transformers 5.20.2 and torch 2.14.1. The tokenizer is saved in the transformers 5 format, so transformers 4.x is not supported.

Reranker results

NDCG@10 when reranking BM25 top-100 (MS MARCO dev: the provided candidates). Measured on an RTX 3060 with the adapter merged into the encoder; with the adapter kept separate, as in this model, rankings match (Spearman 0.999 on a dev sample).

MS MARCO dev is in-domain; the BEIR sets are out-of-domain.

Reranker speed

Documents per second and milliseconds per query when reranking each query's candidates (the same candidate sets as above), on one NVIDIA GeForce RTX 3060. The first query is a warm-up and not counted.

• zev-reranker (this model): as published, default placement (fp32 weights, bf16 autocast). The adapter is kept separate so predict can run without it, which costs about 30% of the throughput (compare the next column). On short passages it is slightly faster than bge-reranker-v2-m3; on the longer BEIR documents it is about as fast.
• zev, adapter merged: the same weights with the adapter merged into a bf16 encoder (rerank only, no decisions); shown for reference.
• MiniLM and bge run in fp16 via sentence-transformers. BEIR documents are longer than MS MARCO passages, hence the lower throughput there.

Limitation: the reranker's null threshold does not transfer

The reranker's null decision is max(score) < null_threshold (model.config.reranker["null_threshold"] = 0.0118). The threshold was learned on MS MARCO and does not transfer to other domains: out of domain, scores shift down as a whole, so the model says "null" for about half of the queries that do have a relevant candidate. The same shift appears for non-English and cross-lingual queries. Ranking is not affected.

Null decisions on the same candidate sets. A query counts as "no relevant candidate" when none of its candidates is in the qrels.

• AUC: how well the best candidate's score separates queries with and without a relevant candidate (0.5 = chance). The signal is there out of domain too, weaker on FiQA.
• accuracy, never null: a model that always answers "something is relevant"; the bar to beat.
• threshold tuned on this set: the best accuracy one threshold reaches on that dataset, with the threshold in brackets. It is tuned on the evaluation data itself, so it is an upper bound for what calibration can give, not a held-out result. The thresholds differ a lot between domains, which is why one shipped threshold cannot serve all of them.
• Scores for this table were computed with the adapter merged into the encoder; rankings match this model.

Before relying on null in a new domain, calibrate the threshold on ~100 labeled queries from that domain and pass it per call (rerank(..., null_threshold=t)):

# labeled: list of (query, docs, has_relevant: bool)
maxes = [(model.score(q, d).max().item(), y) for q, d, y in labeled]
cands = sorted(m for m, _ in maxes)
t = max(cands, key=lambda t: sum((m > t) == y for m, y in maxes))

Without calibration, ignore null and use the ranking only.

Training (reranker)

• The base weights (revision 7b928d8) are frozen and shipped unchanged; only the adapter and the head are trained.
• LoRA r=16, alpha=32 on Wi, Wo, Wqkv in every encoder layer; MLP scoring head plus a learned null logit.
• Listwise softmax cross-entropy with null as an extra candidate (target for queries without a positive), plus KL distillation from BAAI/bge-reranker-v2-m3 (T=2).
• Data: MS MARCO with TAS-B hard negatives (46,441 queries, 18% without a positive), filtered with the teacher to remove likely false negatives. 2 epochs, max length 512. English only.

License

Apache License 2.0 (LICENSE); the mmBERT-base encoder is MIT-licensed (LICENSE-mmBERT). Attributions in NOTICE. The reranker's training data, MS MARCO, is licensed for non-commercial research use only. Review its terms before any commercial use.