Hebrew QA Retriever (fine-tuned multilingual-e5-large)

Retrieves the Hebrew paragraph that answers a Hebrew question. Fine-tuned from intfloat/multilingual-e5-large on the 19,654 answerable training questions of HeQ.

This is the retriever of a three-stage Hebrew RAG system: hybrid search with this model and BM25, then a fine-tuned reranker, then an LLM that writes the answer. Full project, notebooks and demo: GitHub · Demo

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("ElyKok/hebrew-qa-retriever-e5-large-nofilter")
q = model.encode("query: מי ניסח את מגילת העצמאות?", normalize_embeddings=True)
p = model.encode(["passage: ..."], normalize_embeddings=True)
scores = p @ q

Use the query: and passage: prefixes, as with the base model. Works best on paragraphs of about 100-200 words.

Training

• Loss: CachedMultipleNegativesRankingLoss: in-batch negatives plus 2 mined hard negatives per question, batch size 64, gradient caching on a single T4.
• Hard negatives: for each question, the untrained base model retrieves the most similar training paragraphs, and 2 of the top 15 (other than the correct one) are chosen at random.
• One epoch, learning rate 1e-5, 10% warmup, no-duplicates batch sampler.
• "nofilter": unlike a common practice, no candidates were excluded as possible false negatives. In a controlled comparison (same data, seed and settings), excluding same-article and higher-ranked paragraphs lowered R@1 from 80.7 to 78.9.

Results

HeQ test set: 1,072 questions, each searched against all 4,930 paragraphs. Test articles are disjoint from training articles.

Limitations

HeQ questions were written by annotators looking at the paragraph, so they share more wording with it than real user queries do. Training and test data come from Hebrew Wikipedia and Geektime, so gains on other kinds of Hebrew text (for example rabbinic or literary Hebrew) are likely smaller.

HeQ was created by Webiks for MAFAT and is released under CC BY 4.0.