Whisper Small - Fine-tuned for Egyptian Arabic ASR

This repository contains a fine-tuned version of the openai/whisper-small model for Automatic Speech Recognition (ASR) specifically targeting the Egyptian Arabic dialect.

The model was fine-tuned using the SpeechBrain toolkit on the MAdel121/arabic-egy-cleaned dataset.

Model Description

• Base Model: openai/whisper-small
• Language: Arabic (ar)
• Task: Transcription
• Fine-tuning Framework: SpeechBrain
• Dataset: MAdel121/arabic-egy-cleaned

Intended Uses & Limitations

This model is intended for transcribing speech in the Egyptian Arabic dialect.

Limitations:

• Performance may degrade significantly on other Arabic dialects.
• Performance on noisy audio may vary, as only specific augmentations (DropChunk, DropFreq, DropBitResolution) were used during training.
• The model might perform less effectively on highly specialized domains or topics not present in the fine-tuning dataset.

How to Use

You can use this model directly with the transformers library pipeline for automatic speech recognition. Ensure you have transformers and torch installed (pip install transformers torch).

from transformers import pipeline
import torch

# Ensure you have ffmpeg installed for audio processing
# pip install -U ffmpeg-python # or install via system package manager

device = "cuda:0" if torch.cuda.is_available() else "cpu"

# Replace "your-username/whisper-small-egyptian-arabic" with the actual model ID on the Hub
pipe = pipeline(
    "automatic-speech-recognition",
    model="your-username/whisper-small-egyptian-arabic", # <<< Replace this
    device=device
)

# Load your audio file (requires ffmpeg)
# For local files:
audio_file = "/path/to/your/egyptian_arabic_audio.wav"
result = pipe(audio_file, chunk_length_s=30, batch_size=8) # Adjust batch_size based on GPU memory

# For datasets library audio:
# from datasets import load_dataset
# ds = load_dataset("MAdel121/arabic-egy-cleaned", "default", split="test") # Example
# sample = ds[0]["audio"]
# result = pipe(sample.copy()) # Pass a copy to avoid modifying original

print(result["text"])

# --- Using AutoModelForSpeechSeq2Seq ---
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torchaudio

# Load the processor and model (replace with your model ID)
model_id = "your-username/whisper-small-egyptian-arabic" # <<< Replace this with your dataset file on hugging face
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to(device)

# Load and preprocess audio
waveform, sample_rate = torchaudio.load(audio_file)
if sample_rate != processor.feature_extractor.sampling_rate:
    resampler = torchaudio.transforms.Resample(sample_rate, processor.feature_extractor.sampling_rate)
    waveform = resampler(waveform)

input_features = processor(waveform.squeeze().numpy(), sampling_rate=processor.feature_extractor.sampling_rate, return_tensors="pt").input_features.to(device)

# Generate transcription
# Set forced_decoder_ids for Arabic transcription
forced_decoder_ids = processor.get_decoder_prompt_ids(language="ar", task="transcribe")
predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription[0])

Note: The original checkpoint was saved using SpeechBrain. This README assumes the model has been converted to the standard Hugging Face Transformers format for hosting and usage with the pipeline or AutoModel classes. If you are using the original .ckpt file, refer to the project's main README.md and the infer_whisper_local.py script for loading instructions.

Training Data

The model was fine-tuned on the MAdel121/arabic-egy-cleaned dataset available on the Hugging Face Hub. This dataset contains cleaned audio samples and corresponding transcriptions in Egyptian Arabic.

Training Procedure

• Framework: SpeechBrain (speechbrain==1.0.3) with Hugging Face Transformers (transformers==4.51.3) and Accelerate (accelerate==0.25.0).
• Base Model: openai/whisper-small
• Dataset: MAdel121/arabic-egy-cleaned
• Epochs: 10
• Optimizer: AdamW (lr=1e-5, weight_decay=0.05)
• LR Scheduler: NewBob (improvement_threshold=0.0025, annealing_factor=0.9, patient=0)
• Warmup Steps: 1000
• Batch Size: 8 (fixed, no dynamic batching)
• Gradient Accumulation: 2 steps (effective batch size: 16)
• Gradient Clipping: Max norm 5.0
• Mixed Precision: Not explicitly mentioned, assumed FP32 or handled by Accelerate/Trainer.
• Augmentation: Enabled (augment_prob_master=0.5, min_augmentations=1, max_augmentations=3) with the following techniques applied randomly from the pool:
  • DropChunk (length: 1600-4800 samples, count: 1-5)
  • DropFreq (count: 1-3)
  • DropBitResolution
• Training Environment: Modal Labs (gpu=A100-40GB)

Evaluation Results

The model was evaluated on the test split of the MAdel121/arabic-egy-cleaned dataset.

WER (Word Error Rate) and CER (Character Error Rate) are reported. Lower is better.

Validation metrics at the end of training (Epoch 10):

• Validation WER: 22.79%
• Validation CER: 16.76%

Citation

If you use this model, please consider citing the original Whisper paper and the dataset used:

@article{radford2023robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2023}
}

@misc{adel_mohamed_2024_12860997,
  author       = {Adel Mohamed},
  title        = {MAdel121/arabic-egy-cleaned},
  month        = jun,
  year         = 2024,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.12860997},
  url          = {https://doi.org/10.5281/zenodo.12860997}
}

@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}

Model Card Authors

[Your Name/Organization Here]

(Based on training run ceeu3g6c)