Six-turn, Wikipedia-grounded dialogue generation in Modern Standard, Egyptian and Maghrebi Arabic
This LoRA adapter fine-tunes Mistral-7B-Instruct-v0.3 on the training split of ShawarmaChats, a benchmark of 30,012 six-turn conversations grounded in Arabic Wikipedia, with each paragraph rendered in Modern Standard Arabic, Egyptian and Maghrebi Arabic. It was released with the paper Shawarma Chats (ArabicNLP 2025). Given a paragraph and a dialect, the model writes a natural two-speaker dialogue (as JSON) that stays faithful to the paragraph.
1,512 dialogues (504 paragraphs × 3 dialects). Grader: GPT-4o rating with the corpus rubric, A = 4 … D = 1.
| Metric | Base | ShawarmaChats LoRA | Δ |
|---|---|---|---|
| ROUGE-L | 0.304 | 0.443 | +0.139 |
| BLEU | 0.182 | 0.286 | +0.104 |
| BERTScore F1 | 0.767 | 0.856 | +0.089 |
| Grader (1–4) | 1.186 | 3.302 | +2.115 |
Grader ratings after fine-tuning: A 791 · B 408 · C 291 · D 22.
Per dialect (base → fine-tuned):
| Dialect | ROUGE-L | BLEU | BERTScore F1 | Grader (1–4) |
|---|---|---|---|---|
| Modern Standard | 0.248 → 0.408 | 0.142 → 0.256 | 0.698 → 0.838 | 1.414 → 3.058 |
| Egyptian | 0.337 → 0.462 | 0.204 → 0.300 | 0.803 → 0.868 | 1.144 → 3.242 |
| Maghrebi | 0.327 → 0.459 | 0.200 → 0.302 | 0.799 → 0.861 | 1.002 → 3.605 |
import json, torch
from datasets import load_dataset
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "mistralai/Mistral-7B-Instruct-v0.3"
adapter_id = "Kamyar-zeinalipour/ShawarmaChats-Mistral-7B-Instruct-v0.3-LoRA"
tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id).merge_and_unload()
# the prompt used in training: instructions for one dialect + the source paragraph
item = load_dataset("Kamyar-zeinalipour/arabic_multi_dialect_dialogue", "dialogues", split="test")[0]
chat = tokenizer.apply_chat_template(
[{"role": "user", "content": item["prompt"]}], tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(chat, return_tensors="pt").to(model.device)
output = model.generate(
**inputs, max_new_tokens=1024, do_sample=True,
temperature=0.8, top_p=0.95, top_k=50, repetition_penalty=1.1,
)
print(item["dialect"])
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
To build prompts for your own paragraphs, use the templates in
pipeline/prompts.py (standard_prompt, egyptian_prompt,
maghrebi_prompt, each with a {text} placeholder). The base model is gated: accept its terms on
its page and run huggingface-cli login.
The adapter is 0.8 GB because LoRA was also applied to embed_tokens and lm_head, which PEFT
stores with the adapter.
| Base model | mistralai/Mistral-7B-Instruct-v0.3 (7B) |
| Data | ShawarmaChats train split: 28,500 dialogues (9,500 paragraphs × 3 dialects) |
| LoRA | r = 64, α = 128, dropout 0.1; q/k/v/o, gate/up/down, embed_tokens, lm_head |
| Schedule | 3 epochs, AdamW, learning rate 1e-4, cosine, no warm-up, weight decay 1e-4 |
| Batch | 2 sequences per GPU, 2 gradient-accumulation steps, packed to 3,500 tokens |
| Other | bf16, DeepSpeed ZeRO-3, Flash Attention 2, gradient clipping 1.0, seed 100 |
| Decoding (evaluation) | temperature 0.8, top-p 0.95, top-k 50, repetition penalty 1.1, ≤ 1,024 new tokens |
Training, inference and evaluation code: GitHub.
| Base model | Params | Adapter | Grader score (base → fine-tuned) |
|---|---|---|---|
| Llama-3.2-1B-Instruct | 1B | ShawarmaChats-Llama-3.2-1B-Instruct-LoRA | 1.003 → 1.103 |
| Llama-3.2-3B-Instruct | 3B | ShawarmaChats-Llama-3.2-3B-Instruct-LoRA | 1.058 → 1.389 |
| Mistral-7B-Instruct-v0.3 | 7B | ShawarmaChats-Mistral-7B-Instruct-v0.3-LoRA (this model) | 1.186 → 3.302 |
| Llama-3.1-8B-Instruct | 8B | ShawarmaChats-Llama-3.1-8B-Instruct-LoRA | 1.440 → 3.172 |
| Mistral-Nemo-Instruct-2407 | 12B | ShawarmaChats-Mistral-Nemo-Instruct-2407-LoRA | 1.313 → 3.345 |
| Mistral-Small-24B-Instruct-2501 | 24B | ShawarmaChats-Mistral-Small-24B-Instruct-2501-LoRA | 1.854 → 3.607 |
@inproceedings{zeinalipour-etal-2025-shawarma,
title = "Shawarma Chats: A Benchmark Exact Dialogue {\&} Evaluation Platter in {E}gyptian, {M}aghrebi {\&} {M}odern {S}tandard {A}rabic{---}{A} Triple-Dialect Feast for Hungry Language Models",
author = "Zeinalipour, Kamyar and
Saad, Mohamed Zaky and
Attafi, Oumaima and
Maggini, Marco and
Gori, Marco",
editor = "Darwish, Kareem and
Ali, Ahmed and
Abu Farha, Ibrahim and
Touileb, Samia and
Zitouni, Imed and
Abdelali, Ahmed and
Al-Ghamdi, Sharefah and
Alkhereyf, Sakhar and
Zaghouani, Wajdi and
Khalifa, Salam and
AlKhamissi, Badr and
Almatham, Rawan and
Hamed, Injy and
Alyafeai, Zaid and
Alowisheq, Areeb and
Inoue, Go and
Mrini, Khalil and
Alshammari, Waad",
booktitle = "Proceedings of The Third {A}rabic Natural Language Processing Conference",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.arabicnlp-main.39/",
doi = "10.18653/v1/2025.arabicnlp-main.39",
pages = "472--524",
ISBN = "979-8-89176-352-4"
}
Derivative of Mistral-7B-Instruct-v0.3, released under the base model's license
(apache-2.0). ShawarmaChats data: CC BY-SA 4.0. Code: Apache-2.0.
mistralai/Mistral-7B-Instruct-v0.3