ettin-reranker-17m-commonplace

A 17M-parameter cross-encoder reranker for offline search on a phone. It is cross-encoder/ettin-reranker-17m-v1, fine-tuned by listwise distillation from cross-encoder/ettin-reranker-400m-v1.

I made it for Commonplace, an offline research assistant for Android. The app searches local copies of Wikipedia, Stack Exchange, Wikibooks, Wikivoyage and other sources. This model reranks the top 24 passages that keyword and vector search find for each question. The architecture and the inference cost are the same as the base model.

Results

The test set has 161 questions: the app's seed questions and the first turns of its test conversations. Each question has a pool of 24 passages from the app's own retrieval. Claude Sonnet graded every passage (0 = not relevant, 1 = relevant, 2 = answers the question). The "explain" subset has the 46 questions that ask why or how.

NQ is the Natural Questions open-domain validation set (2,610 held-out questions). A question is a hit at k when an answer string occurs in one of the top k passages of its pool.

The gain on the 161 questions is small: +1.2 points of nDCG@8 (95% confidence interval +0.1 to +2.3). 76 questions improved and 55 got worse. Do not expect a large change on general web search benchmarks.

Files

The ONNX files contain the transformer only. Each one outputs last_hidden_state. To get the score, take the first token's vector and apply the head from 2_Dense, 3_LayerNorm and 4_Dense: Dense 256 (GELU, no bias) → LayerNorm → Dense 1.

The model was trained with inputs of at most 256 tokens.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("0xPD33/ettin-reranker-17m-commonplace", max_length=256)
scores = model.predict([
    ("How do vaccines train the immune system?", "A vaccine contains an antigen that ..."),
    ("How do vaccines train the immune system?", "The first smallpox vaccine was ..."),
])

Higher scores mean more relevant. The scores are logits, not probabilities.

Training

• Start: cross-encoder/ettin-reranker-17m-v1.
• Teacher: cross-encoder/ettin-reranker-400m-v1 scored every passage of every pool.
• Loss: KL divergence between the teacher's and the student's softmax over each pool, temperature 1. Each step used 16 queries with 16 passages each (the teacher's top 2 and 14 random others).
• Schedule: 3 epochs (4,314 steps), AdamW, learning rate 3e-5, one-cycle with 5% warm-up, weight decay 0.01. I kept the checkpoint at step 3,250, chosen on 1,000 NQ dev questions and 368 held-out generated questions. The 161 test questions were not used for training or checkpoint selection.
• Hardware: one RTX 5060 Ti, 17 minutes.

Data

23,012 queries. Each query has a pool of up to 24 passages from the Commonplace search pipeline (BM25 and dense retrieval, fused):

• 11,475 questions from the NQ-open train split.
• 2,635 Stack Exchange question titles.
• 8,902 generated questions. A local Gemma model wrote two questions per sampled passage. The mix is about 48% "why/how" questions, 17% lookups, 14% comparisons, 8% how-to questions and 12% search queries from Stack Exchange post bodies. The passages come from English Wikipedia (42%), Stack Exchange (20%), Wikibooks (13%), Wikiversity, textbooks, Wikivoyage, arXiv abstracts, Wikiquote, OpenStax, MedlinePlus and others.

A filter removed generated questions that were near-duplicates of the test questions or the NQ dev questions.

Limitations

• English only.
• The model is small. On short or unusual texts it can prefer a passage that repeats the question's words over a passage that answers the question.
• The test set is small (161 questions) and the grades come from an LLM.

License

Apache 2.0, the same as the base model and the teacher. The training queries and passages come from sources under their own licenses (for example CC BY-SA for Wikipedia, Stack Exchange and NQ). This repository does not contain those texts.