Hebrew QA Reranker (fine-tuned bge-reranker-v2-m3)
A cross-encoder that scores how well a Hebrew paragraph answers a Hebrew question. Fine-tuned from
BAAI/bge-reranker-v2-m3 on the answerable questions of HeQ,
with 3 hard negatives per question taken from a hybrid BM25 + fine-tuned e5-large first stage, and
BinaryCrossEntropyLoss (1 epoch, lr 1e-05, 2 x T4).
Use it as the second stage of a search: retrieve the top 20 paragraphs, then rerank them.
from sentence_transformers.cross_encoder import CrossEncoder
model = CrossEncoder("ElyKok/hebrew-qa-reranker")
scores = model.predict([("מי ניסח את מגילת העצמאות?", "paragraph 1"), ("מי ניסח את מגילת העצמאות?", "paragraph 2")])
Results (HeQ test set, 1,072 questions, reranking the top 20 of 4,930 paragraphs)
| Model | R@1 | R@5 | R@10 | MRR@10 | NDCG@10 |
|---|---|---|---|---|---|
| Hybrid (BM25 + e5-large, no training) | 80.4 | 92.1 | 95.1 | 85.5 | 87.8 |
| Hybrid (BM25 + e5-large, no training) + bge-reranker-v2-m3 (no training) | 91.5 | 96.5 | 96.8 | 93.8 | 94.6 |
| Hybrid (BM25 + e5-large, no training) + fine-tuned reranker (ours) | 93.8 | 96.6 | 96.8 | 95 | 95.5 |
| Hybrid (BM25 + fine-tuned e5-large) | 83.3 | 93.4 | 95.5 | 87.6 | 89.5 |
| Hybrid (BM25 + fine-tuned e5-large) + bge-reranker-v2-m3 (no training) | 91.7 | 97.2 | 97.4 | 94.1 | 95 |
| Hybrid (BM25 + fine-tuned e5-large) + fine-tuned reranker (ours) | 93.9 | 97.2 | 97.4 | 95.4 | 95.9 |
Test articles are disjoint from training articles. HeQ is licensed CC BY 4.0 (Webiks / MAFAT).