multilingual-e5-small, fine-tuned for Arabic-English retrieval

A 118M-parameter embedder for search over mixed Arabic and English text, including the hard case: an Arabic query finding an English passage. Fine-tuned from intfloat/multilingual-e5-small with mined hard negatives and cross-lingual training pairs.

Code, benchmark and full results: github.com/mahmoudzahran01/arabic-english-retrieval-eval. Companion reranker: mzahran1/arabic-english-reranker-mminilm.

Results

Test MRR@10 x 100 on a bilingual benchmark built from TyDi-QA, ARCD and XQuAD (6,823 test queries; disjoint train/dev/test articles; leakage-audited). Mean ± std over 3 training seeds (this checkpoint is seed 13, macro 75.7).

Above bge-m3 on the macro average at a fifth of its size, and +12.6 on en->ar; bge-m3 is still better on ar->en alone (63.3 vs 54.7), and the reranker closes that gap.

Usage

The e5 prefixes are required: query: for queries and passage: for documents.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("mzahran1/arabic-english-e5-small-retrieval")
query = model.encode(["query: متى تأسست جامعة الملك عبدالله للعلوم والتقنية؟"])
docs = model.encode([
    "passage: The King Abdullah University of Science and Technology was founded in 2009 in Thuwal.",
    "passage: Riyadh is the capital of Saudi Arabia and its largest city.",
])
print(model.similarity(query, docs))  # the KAUST passage scores highest

Training

• Data: 16,419 monolingual training queries (TyDi-QA Arabic and English, ARCD, SQuAD) plus LLM-translated cross-lingual queries (ar->en and en->ar), 20,918 in total. XQuAD is test-only.
• Negatives: 1 hard negative per query, mined with the base model from ranks 1-30, with a 0.02-cosine margin filter against false negatives (an LLM audit found 8.8% of top-3 mined negatives actually contain the answer).
• Recipe: MultipleNegativesRankingLoss, batch 64 (no-duplicates sampler), lr 3e-5, 10% warm-up then linear decay, scale 20, 1 epoch, max 256 tokens, bf16, on one NVIDIA GB10.

Limitations

• Evaluated on Wikipedia-style factoid QA with one gold passage per query; gains on enterprise documents are not measured.
• The cross-lingual training queries are machine-translated, so they are cleaner than real code-mixed Arabic.
• English retrieval is unchanged (87.8 -> 87.3 macro en->en); the model trades a little English for Arabic and cross-lingual quality.
• Text beyond 256 tokens was not seen in training.

License

MIT, like the base model. The training data keeps its own licences (TyDi-QA Apache-2.0, ARCD MIT, SQuAD CC BY-SA 4.0).