whisper-medium-arabic-dialectal

Fine-tune of openai/whisper-medium (769M) for multi-dialect Arabic speech recognition (undiacritized output).

Private / internal model. Evaluate on your own data before production use.

Results (932-clip held-out test set)

Kept improving to ~epoch 4.8. ~50% relative WER cut over baseline.

Model comparison — all Arabic ASR models

Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags, keep punctuation + dialect spelling) — so every row is directly comparable.

• Best fine-tuned: whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic a close second (0.357).
• Best zero-shot: cohere-transcribe-arabic (0.457, Arabic-specialized). A full fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning with best-checkpoint selection was the fix.
• Streaming / low-latency: nemotron-3.5-asr.
• Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining; cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).

Dataset

Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).

• ~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
• Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi (Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
• Augmented: each clean clip is expanded with variants (noise, music, speed, voice, reverb/codec) via an augmentation column. Derived from oddadmix/dialectal-arabic-lahgtna-v2-smaller.

Preprocessing (applied to targets)

• Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags ([laughter], [exhale], [inhale], [mumble], [cough], timestamps, …).
• Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and sentence punctuation. Output is undiacritized.
• Filtered: dropped clips > 30 s (~10%; Whisper's encoder limit) and a few corrupt rows (12k-token transcripts). ~62% of rows had harakat, ~28% had non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.

Training

• Base: openai/whisper-medium · full fine-tune (no LoRA)
• Effective batch 32 · LR 1e-5 (warmup 500, cosine) · 6000 steps · best @ step 5500 · ~183 min
• Hardware: a single 32 GB GPU, bf16 autocast (fp32 weights + generate)
• Metric: WER/CER on cleaned references (clean-text normalized)

Fine-tuning (reproduce)

The exact fine-tuning code is bundled in this repo (train.py, normalize.py, evaluate_model.py) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).

pip install -r requirements.txt
huggingface-cli login          # for the (private) dataset

python train.py \
  --base_model openai/whisper-medium --run_name my-run \
  --per_device_train_batch_size 8 --gradient_accumulation_steps 4 \
  --learning_rate 1e-5 --warmup_steps 500 --max_steps 6000

python evaluate_model.py --model runs/my-run          # WER / CER

Note: some checkpoints (e.g. large-v3-turbo) ship in fp16 — train.py force-loads fp32 so generate() doesn't crash at eval. bf16 autocast is used for training.

Usage

import torch, torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor

repo = "oddadmix/whisper-medium-arabic-dialectal"
proc = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.float16).to("cuda").eval()

wav, sr = torchaudio.load("clip.wav")          # resample to 16 kHz mono if needed
feats = proc(wav.mean(0).numpy(), sampling_rate=16000,
             return_tensors="pt").input_features.to("cuda", torch.float16)
ids = model.generate(feats, language="ar", task="transcribe")
print(proc.batch_decode(ids, skip_special_tokens=True)[0])

Learnings & notes

• Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags (rather than aggressive letter-folding) was key for a multi-dialect model.
• Whisper fine-tunes furthest; the offline (full-context) models beat the streaming Nemotron on pure WER, though Nemotron has the best zero-shot (native Arabic) and offers streaming.
• Blackwell (sm_120) GPU gotchas for the NeMo/Nemotron model: numba's RNNT loss needs numba==0.61.2 + NUMBA_CUDA_USE_NVIDIA_BINDING=1 + cuda-python<13 to emit sm_120 code, and fp32 (the numba loss can't take bf16). Nemotron 3.5 requires NeMo main (its EncDecRNNTBPEModelWithPrompt class isn't in any release) and a per-sample lang=ar prompt.

Limitations

• Trained on augmented, partly synthetic multi-dialect data; real-world dialect coverage varies (Maghrebi is the hardest).
• Output is undiacritized and lower-cased for Latin tokens.
• Offline model (≤30 s chunks); not streaming.