Jisr-MT-50M-Dialects2En
13 Arabic dialects → English, one 49M-parameter model.
+12.63 chrF++ over its base on every dialect tested, reaching 76.2% of the 12B model that generated its training data — at 1/245th the size.
Results
UBC-NLP/alexandria test splits, beam 4. gemma-4-12B-it is the model that produced this
model's training targets, so it is the effective teacher ceiling.
| dialect | base 49M | this model | gain | gemma-4-12B | % of teacher |
|---|---|---|---|---|---|
| Syrian | 37.19 | 51.48 | +14.29 | 64.52 | 79.8% |
| Sudanese | 35.38 | 47.79 | +12.40 | 57.60 | 83.0% |
| Saudi | 38.90 | 47.53 | +8.64 | 63.16 | 75.3% |
| Lebanese | 31.11 | 46.34 | +15.23 | 55.72 | 83.2% |
| Tunisian | 32.34 | 46.19 | +13.85 | 64.42 | 71.7% |
| Palestinian | 34.33 | 46.15 | +11.81 | 55.71 | 82.8% |
| Egyptian | 32.72 | 45.11 | +12.39 | 61.08 | 73.9% |
| Yemeni | 34.34 | 43.88 | +9.55 | 57.36 | 76.5% |
| Libyan | 32.65 | 41.50 | +8.85 | 53.17 | 78.1% |
| Moroccan | 25.71 | 40.60 | +14.89 | 55.69 | 72.9% |
| weighted avg | 32.66 | 45.29 | +12.63 | 59.46 | 76.2% |
Weighted by each dialect's share of the training corpus.
Why this direction works so much better than English → dialect
The same corpus trains a worse model in the reverse direction. For English → Egyptian it scored 34.96 chrF++ against 37.91 for a purpose-built human corpus.
The reason is which side is machine-generated. Here the source is genuine human speech (ASR transcripts) and the target is Gemma's English. Noise on the source side costs far less than noise on the target side, because the model is never asked to reproduce it. Going the other way, the model had to generate machine-translated Arabic.
This makes the run effectively sequence-level distillation of a 12B model into 49M — the same mechanism that let Jisr-15M-distill reach 83.5% of a 241M teacher.
Compared with frontier models
Same benchmark, 200 turns per dialect. gemma-4-12B-it produced this model's training targets;
gpt-5.1 never saw this corpus and is an independent read.
| dialect | this model (49M) | gpt-5.1 | gemma-4-12B | % of gpt-5.1 |
|---|---|---|---|---|
| Lebanese | 46.34 | 53.16 | 55.72 | 87% |
| Palestinian | 46.15 | 53.25 | 55.71 | 87% |
| Sudanese | 47.79 | 55.23 | 57.60 | 87% |
| Syrian | 51.48 | 59.80 | 64.52 | 86% |
| Libyan | 41.50 | 49.55 | 53.17 | 84% |
| Yemeni | 43.88 | 52.86 | 57.36 | 83% |
| Saudi | 47.53 | 57.68 | 63.16 | 82% |
| Egyptian | 45.11 | 56.44 | 61.08 | 80% |
| Tunisian | 46.19 | 59.44 | 64.42 | 78% |
| Moroccan | 40.60 | 52.61 | 55.69 | 77% |
| weighted avg | 45.29 | 55.48 | 59.46 | 82% |
A 49M model reaches 82% of gpt-5.1, within 7.4–12 chrF++ of it on every dialect, after 23 minutes of training on one GPU. The spread is tight (77–87%), unlike the English→dialect direction where the same family ranges more widely.
Two things worth noting
gpt-5.1 scores below gemma-4-12B on all 10 dialects (55.48 vs 59.46 weighted). Gemma generated this project's training English, so its style sits closer to conventional MT English — but Alexandria's English references are human-written and independent of that corpus, so this is a real result on a neutral benchmark rather than contamination. Treat it as surprising and worth a second look, not settled.
Moroccan is unremarkable here (gpt-5.1 52.61, mid-pack) while in the English→Moroccan direction gpt-5.1 collapses to 20.86 — below the 49M Jisr-MT-50M-Moroccan at 32.23. Frontier models can understand Darija; they struggle to produce it.
⚠️ It costs MSA
FLORES+ devtest, MSA ar→en, 1,012 sentences:
| model | BLEU | chrF | chrF++ |
|---|---|---|---|
| Jisr-MT-50M (base) | 29.35 | 57.86 | 55.87 |
| this model | 23.18 | 52.59 | 50.65 |
| −6.17 | −5.27 | −5.22 |
The corpus included MSA→English replay at 0.3×, and it was not enough. The reasoning behind that ratio — that forgetting would be milder because the target language does not change — was only partly right: the model still lost 6.17 BLEU of MSA.
Use the base model for Modern Standard Arabic. Use this one for dialect. A future run should try 1× replay, which is what held MSA in place in the English→dialect models.
Usage
from transformers import AutoTokenizer, MarianMTModel
tok = AutoTokenizer.from_pretrained("oddadmix/Jisr-MT-50M-Dialects2En")
model = MarianMTModel.from_pretrained("oddadmix/Jisr-MT-50M-Dialects2En")
text = ">>eng<< العيال بيلعبوا كورة في الشارع بعد المدرسة"
out = model.generate(**tok(text, return_tensors="pt"), num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))
One tag, >>eng<<, and it is required. There is no per-dialect tag: the target is English
for every input, so the model infers the dialect from the text itself. Arabic → English only —
this model has no English → Arabic direction.
Training
| Base | Jisr-MT-50M (49M, ar→en 29.35 BLEU) |
| Dialect data | 1,272,744 pairs across 13 dialects |
| MSA replay | 381,823 ar→en pairs (0.3×) — too little, see above |
| Total | 1,654,567 rows, single >>eng<< tag |
| Budget | 1 epoch, 25,852 steps, bs 64, lr 3e-4 cosine, bf16 |
| Hardware | 23 minutes |
The base is Jisr-MT-50M, not Jisr-MT-50M-tatoeba. The tatoeba model has the family's best
en→ar but its ar→en is broken (7.86 BLEU), and this direction depends on it entirely.
All 13 dialects were pooled into one model rather than trained separately, following the AllDialects result where small dialects gained most from sharing (Libyan +7.62 chrF++).
Limitations
- MSA regressed by 6.17 BLEU — see above.
- Arabic → English only.
- Algerian, Bahraini and Iraqi are unvalidated.
UBC-NLP/alexandriahas no config for them, so 3 of the 13 dialects this model claims have no external benchmark at all. - Cannot exceed its teacher; it reaches 76.2% of
gemma-4-12B-itand inherits its biases. - Source register is spoken/conversational — the training data is ASR transcripts.
- 49M parameters: expect errors on rare named entities and long complex input.