加载中…
加载中…
language:
Arabic laws, regulations, and other formal documents are usually written in Modern Standard Arabic (MSA). However, users often ask questions in their local dialect.
This difference creates a challenge for semantic retrieval. An embedding model may retrieve the correct MSA passage for an MSA query but perform worse when the same query is written in a dialect.
e5-small-dialect2MSA addresses this problem. It is a fine-tuned version of multilingual-e5-small, trained to improve the retrieval of MSA passages from queries written in MSA or Arabic dialects.
The model supports Saudi, Syrian, Egyptian, Sudanese, and Tunisian Arabic.
The main objective is to retrieve relevant MSA content directly from dialect queries without requiring query rewriting or normalization.
We compare the base model with the fine-tuned model on 1,238 queries per language. The evaluation measures whether relevant MSA content appears in the top five retrieved results and how highly it ranks.
The results show improvements for MSA and all five supported dialects.
Mean Reciprocal Rank at 5 (MRR@5) measures the ranking position of the first relevant passage in the top five results. Higher values indicate better ranking quality.
| Query language | Base model | Fine-tuned model | Improvement (pp) |
|---|---|---|---|
| MSA | 0.8411 | 0.8830 | +4.19 |
| Saudi | 0.7451 | 0.8563 | +11.12 |
| Syrian | 0.7460 | 0.8590 | +11.30 |
| Egyptian | 0.7531 | 0.8556 | +10.25 |
| Sudanese | 0.7241 | 0.8485 | +12.44 |
| Tunisian | 0.6542 | 0.8265 | +17.23 |
Recall at 5 measures the proportion of queries for which at least one relevant passage appears in the top five results. Higher values indicate better retrieval coverage.
| Query language | Base model | Fine-tuned model | Improvement (pp) |
|---|---|---|---|
| MSA | 0.9596 | 0.9830 | +2.34 |
| Saudi | 0.9273 | 0.9798 | +5.25 |
| Syrian | 0.9233 | 0.9790 | +5.57 |
| Egyptian | 0.9346 | 0.9774 | +4.28 |
| Sudanese | 0.9184 | 0.9750 | +5.66 |
| Tunisian | 0.8708 | 0.9596 | +8.88 |
1. Better retrieval from Arabic dialects
The fine-tuned model improves both metrics for every supported dialect. This shows that it is better at matching dialect queries with relevant MSA passages.
2. The largest gains occur with Tunisian Arabic
Tunisian queries show the largest improvement in both metrics:
This is particularly important because Tunisian Arabic had the lowest baseline scores.
3. The gap between MSA and dialect retrieval becomes smaller
The base model performs better on MSA queries than on dialect queries. Fine-tuning improves the scores for all dialects and reduces this performance gap.
4. MSA retrieval also improves
The fine-tuned model improves MRR@5 from 0.8411 to 0.8830 and Recall@5 from 0.9596 to 0.9830 for MSA queries.
The fine-tuning therefore improves retrieval for both the original query language and its dialect variants.
These results apply to the current evaluation dataset and setup. Performance on other corpora or real-world queries may differ.
A legal AI system often stores laws and regulations in MSA, while users may ask questions in their local dialect.
For example, a user may ask a legal question in Tunisian Arabic, while the relevant law is written in formal MSA.
With a standard embedding model, the system may need to rewrite the question into MSA before retrieval. This adds an extra step and may change the meaning of the question.
With OussamaBenSlama/e5-small-dialect2MSA, the system can encode the original dialect query and retrieve relevant MSA passages directly.
This approach reduces the need for query rewriting and focuses on direct dialect-to-MSA retrieval.
| Property | Description |
|---|---|
| Model | OussamaBenSlama/e5-small-dialect2MSA |
| Base model | intfloat/multilingual-e5-small |
| Task | Semantic retrieval |
| Query languages | MSA, Saudi, Syrian, Egyptian, Sudanese, Tunisian |
| Target content | MSA text chunks |
| Training method | Contrastive fine-tuning with hard negatives |
The training data starts with original MSA queries. Each query is translated into five dialect variants.
The dataset therefore contains queries in six forms:
| Text column | Language |
|---|---|
query |
MSA |
saudi_variant |
Saudi |
syrian_variant |
Syrian |
egyptian_variant |
Egyptian |
sudanese_variant |
Sudanese |
tunisian_variant |
Tunisian |
We use a two-stage translation pipeline to generate dialect variants:
facebook/nllb-200-1.3Boddadmix/Jisr-MT-50M-AllDialectsEnglish is used as an intermediate language between MSA and the target dialect.
This approach allows us to generate dialect variants using relatively small translation models and limited computing resources.
Each training example contains:
The model learns to place a query closer to its positive passage than to the hard negatives.
This training setup aims to improve the similarity between dialect queries and their corresponding MSA content.
The project uses the following datasets as data sources:
The cleaned and translated training, validation, and test data is available in the Hugging Face dataset repositories linked at the top of this README.
Install the required library:
pip install sentence-transformers
Load the model and compare a dialect query with an MSA passage:
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer(
"OussamaBenSlama/e5-small-dialect2MSA"
)
query = "كيفاش نجم نعمل شكاية؟" # Tunisian Arabic
passage = "تحدد القوانين الإجراءات اللازمة لتقديم الشكاوى."
query_embedding = model.encode(
f"query: {query}",
normalize_embeddings=True
)
passage_embedding = model.encode(
f"passage: {passage}",
normalize_embeddings=True
)
score = cos_sim(query_embedding, passage_embedding)
print(score.item())
The original queries have an average length of approximately 13 tokens.
Short queries may not cover the range of questions found in real applications, including long questions, complex wording, and questions with several parts.
The dialect variants are generated through machine translation:
MSA → English → Arabic dialect.
Translation errors may change the meaning of a query or produce unnatural dialect expressions. This can affect the quality of the training data and the final model.
The translation models were selected partly because of limited computing resources. More powerful translation models or human-written dialect queries may improve the data quality and retrieval performance.
The model supports five dialects: Saudi, Syrian, Egyptian, Sudanese, and Tunisian.
Performance on other Arabic dialects, mixed dialects, or other language varieties has not been established by the current evaluation.
Potential improvements include: