Whisper-small: Tunisian Derja (with French/English code-switching)

This is openai/whisper-small (242 M parameters), fully fine-tuned to transcribe Tunisian Arabic (Derja), including the French and English words Tunisians mix into everyday speech.

• This repo: Transformers weights (fp32 safetensors, 967 MB).
• Faster / CPU / real-time use: the CTranslate2 version, TheBiggyBang/whisper-small-tunisian-derja-ct2 (250 MB, int8).
• Code, training notebook and full evaluation: GitHub

Results

Held-out test set: 482 clips (36.6 min) drawn from all LinTO test splits. No recording in it was seen during training or checkpoint selection.

• Normalized WER removes Derja spelling variants that are not recognition errors: أ/إ/آ→ا, ى→ي, ة→ه, Arabic-Indic digits, punctuation, lowercase Latin, and standalone «و» attached to the next word. The same normalization is applied to references and hypotheses.
• 95 % confidence intervals come from a cluster bootstrap over recordings.
• Against Vosk: this model makes fewer character errors (CER 12.0 vs 15.7) but more word errors. Its largest error category (about 31 %) is near-miss spelling (see Limitations).

Examples from the test set:

How to use

import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

repo = "TheBiggyBang/whisper-small-tunisian-derja"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo)

audio, _ = librosa.load("audio.wav", sr=16000)          # mono, 16 kHz, up to 30 s
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features, language="arabic", task="transcribe", num_beams=3)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

For recordings longer than 30 s, use the pipeline with chunking:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="TheBiggyBang/whisper-small-tunisian-derja", chunk_length_s=30)
print(asr("long_audio.wav", generate_kwargs={"language": "arabic", "task": "transcribe"})["text"])

For speed, CPU deployment or live microphone transcription, use the CTranslate2 version.

Output style: Derja in Arabic script. French/English words appear in Latin script (or Arabic script, as the annotators sometimes wrote them). Punctuation is kept and diacritics are removed.

Training data

Trained on the LinTO Tunisian datasets by LINAGORA, both licensed CC-BY-4.0. The default config of each was streamed (all sub-corpora):

These include TunSwitch CS/TO, MASC, AmenyKH, ApprendreLeTunisien, OneStory, Tunisian_dataset_STT-TTS15s, Wav2Vec-tunisian-Darja and several YouTube collections. See the dataset cards for each source.

Leakage controls:

• Every evaluation sentence with ≥ 3 words (679 sentences) is block-listed. Training recordings that contain one are dropped (97 recordings).
• Dev and test are split by recording, not by segment.

Training procedure

Evaluation protocol

• Pool: all six original test splits (AmenyKH, ApprendreLeTunisien, OneStory, TunSwitchCS, TunSwitchTO, Youtube_TNScrapped_V1).
• Split: by recording into dev (323 clips, used for checkpoint selection) and test (482 clips, used once).
• Test composition: TunSwitchCS 185 · TunSwitchTO 171 · ApprendreLeTunisien 69 · AmenyKH 30 · Youtube_TNScrapped_V1 15 · OneStory 12.
• Metrics are corpus-level (total edits / total reference words).

Limitations

• Speaker overlap is possible. The datasets have no speaker IDs, so the test set is recording-disjoint but not speaker-disjoint. Real-world accuracy on new speakers may be lower.
• Spelling. Derja has no standard orthography. The largest error category is near-miss spelling (e.g. لي/الي, موش/مش, والا/ولا), so report CER alongside WER.
• Code-switching. French/English words may come out in a different script than a human annotator would use.
• Small test set (36.6 min). Per-corpus numbers are not reported because some sub-corpora have only 12–15 test clips.
• Inherited Whisper risks. Hallucinations and repetition are possible on noisy or unclear audio. Do not use it on recordings made without consent or for high-stakes decisions.

Attribution

Fine-tuned from openai/whisper-small (Apache-2.0). Training data: the LinTO DataSet Audio for Arabic Tunisian and its augmented version by LINAGORA, licensed under CC-BY-4.0. These include TunSwitch (Abdallah et al., 2023), MASC (Al-Fetyani et al., 2021) and other community and web sources listed on the dataset cards. The data was segmented, filtered and used for training; no audio or transcripts are redistributed here.

Citation

If you use this model, please cite the works it builds on:

@misc{linagora2024Linto-tn,
  title = {LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},
  author = {Hedi Naouara and Jérôme Louradour and Jean-Pierre Lorré},
  year = {2025},
  month = {March},
  eprint={2504.02604},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  note={Good Data Workshop, AAAI 2025},
  url={arxiv.org/abs/2504.02604},
}

@misc{abdallah2023leveraging,
      title={Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition},
      author={Ahmed Amine Ben Abdallah and Ata Kabboudi and Amir Kanoun and Salah Zaiem},
      year={2023},
      eprint={2309.11327},
      archivePrefix={arXiv},
      primaryClass={eess.AS}
}

@data{e1qb-jv46-21,
doi = {10.21227/e1qb-jv46},
url = {https://dx.doi.org/10.21227/e1qb-jv46},
author = {Al-Fetyani, Mohammad and Al-Barham, Muhammad and Abandah, Gheith and Alsharkawi, Adham and Dawas, Maha},
publisher = {IEEE Dataport},
title = {MASC: Massive Arabic Speech Corpus},
year = {2021} }

@misc{radford2022whisper,
  doi = {10.48550/ARXIV.2212.04356},
  url = {https://arxiv.org/abs/2212.04356},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}