Whisper Small - Urdu Fine-tuned

This model is a fine-tuned version of openai/whisper-small for Urdu (اردو) automatic speech recognition (ASR), trained on the expanded Mozilla Common Voice Scripted Speech 24.0 - Urdu dataset, accessed via the Mozilla Data Collective API.

Model Description

• Developed by: Khawaja Ali
• Model type: Whisper (Encoder-Decoder Transformer)
• Language: Urdu (ur)
• License: Apache 2.0
• Finetuned from: openai/whisper-small
• Best Checkpoint: Step 3500 (loaded via load_best_model_at_end)
• W&B Run: whisper-small-urdu-expanded-v1

Intended Uses & Limitations

Intended Uses

• Transcribing Urdu speech to text
• Building Urdu voice assistants
• Subtitling Urdu audio/video content
• Accessibility applications for Urdu speakers

Limitations

• Domain: Trained on read speech; may underperform on conversational/spontaneous speech
• Accents: Primarily trained on Common Voice contributors; dialect coverage may vary
• Noise: Best performance on clean audio; background noise degrades accuracy
• Audio format: Requires 16kHz mono audio

Performance

Baseline Comparison (Base Models, Zero-Shot on Urdu):

🎉 This fine-tuned small model outperforms whisper-medium and approaches whisper-large-v2 performance!

Quick Start

Installation

pip install transformers torch librosa

Inference with Pipeline (Recommended)

from transformers import pipeline
import torch

device = "cuda:0" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
    "automatic-speech-recognition",
    model="khawajaaliarshad/whisper-small-urdu",
    device=device,
)

# Transcribe audio file
result = pipe(
    "audio.wav",
    generate_kwargs={"language": "urdu", "task": "transcribe"},
)
print(result["text"])

Inference with Transformers

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch
import librosa

# Load model
processor = WhisperProcessor.from_pretrained("khawajaaliarshad/whisper-small-urdu")
model = WhisperForConditionalGeneration.from_pretrained("khawajaaliarshad/whisper-small-urdu")

device = "cuda:0" if torch.cuda.is_available() else "cpu"
model.to(device)

# Load and process audio
audio, sr = librosa.load("audio.wav", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device)

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(
        inputs.input_features,
        language="urdu",
        task="transcribe",
        max_length=225,
    )

# Decode
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Long-form Transcription

For audio longer than 30 seconds:

result = pipe(
    "long_audio.wav",
    chunk_length_s=30,
    batch_size=8,
    generate_kwargs={"language": "urdu", "task": "transcribe"},
)
print(result["text"])

Training Details

Training Data

The model was trained on the expanded Common Voice Scripted Speech 24.0 - Urdu dataset downloaded via the Mozilla Data Collective API.

Processed splits used for training (expanded):

Processed Dataset: khawajaaliarshad/common-voice-urdu-processed-expanded

This is the preprocessed version of the dataset used for training, with audio resampled to 16kHz and ready for Whisper fine-tuning.

Note: The model was trained on clean (non-augmented) data. Initial experiments with audio augmentation showed degraded performance due to train/eval domain mismatch.

Training Hyperparameters

Training Infrastructure

Training Progress

Note: Best WER was achieved at step 3500 (32.80%). The model loaded via load_best_model_at_end=True uses this checkpoint.

Framework Versions

• Python: 3.12
• Transformers: 4.52.0
• Datasets: 2.20.0
• PyTorch: 2.8.0
• Accelerate: 0.30+

Evaluation

Metrics

• WER (Word Error Rate): Primary metric for ASR evaluation
• Computed using jiwer library

Evaluation Code

from datasets import load_dataset
from transformers import pipeline
from evaluate import load

# Load test data
test_ds = load_dataset("mozilla-foundation/common_voice_17_0", "ur", split="test")

# Load model
pipe = pipeline("automatic-speech-recognition", model="khawajaaliarshad/whisper-small-urdu")

# Run evaluation
wer_metric = load("wer")
predictions, references = [], []

for sample in test_ds:
    result = pipe(sample["audio"]["array"], generate_kwargs={"language": "urdu"})
    predictions.append(result["text"])
    references.append(sample["sentence"])

wer = wer_metric.compute(predictions=predictions, references=references)
print(f"WER: {wer * 100:.2f}%")

Environmental Impact

Training was performed on Kaggle's free GPU tier, utilizing shared infrastructure.

Estimated using ML CO2 Impact Calculator

Citation

BibTeX

If you use this model in your research, please cite:

@misc{khawaja2025whisperurdu,
  author = {Khawaja, Ali},
  title = {Whisper Small Fine-tuned for Urdu Speech Recognition},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/khawajaaliarshad/whisper-small-urdu}},
}

Please also cite the original Whisper paper:

@article{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}

And the Mozilla Common Voice dataset:

@inproceedings{ardila2020common,
  title={Common Voice: A Massively-Multilingual Speech Corpus},
  author={Ardila, Rosana and Branson, Megan and Davis, Kelly and Henretty, Michael and Kohler, Michael and Meyer, Josh and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M and Weber, Gregor},
  booktitle={Proceedings of the 12th Language Resources and Evaluation Conference},
  pages={4211--4215},
  year={2020}
}

Data Access: The Urdu dataset was accessed via the Mozilla Data Collective API (Dataset ID: cmj8u3pz600t9nxxbz9l2ck2n). The data is licensed under CC0-1.0 (Public Domain).

Acknowledgments

This project was made possible by the following resources and communities:

Models & Data

• OpenAI Whisper - For the foundational multilingual speech recognition model
• Mozilla Common Voice - For the open-source Urdu speech dataset contributed by volunteers worldwide
• Mozilla Data Collective - For the API access to download Common Voice datasets
• Mozilla Foundation - For supporting open speech data initiatives
• Hugging Face - For the transformers library and model hosting infrastructure

Compute & Tools

• Kaggle - For providing free GPU compute (T4)
• Weights & Biases - For experiment tracking and visualization
• Google Colab - For initial experimentation

Libraries

• 🤗 Transformers
• 🤗 Datasets
• 🤗 Evaluate
• 🤗 Accelerate
• jiwer - WER computation
• librosa - Audio processing

Community

• The Urdu-speaking contributors to Mozilla Common Voice
• The Hugging Face community for guidance on Whisper fine-tuning
• The open-source ML community

Model Card Authors

• Khawaja Ali - @khawajaaliarshad

Contact

For questions, issues, or collaboration:

• 🐛 Issues: Open an issue on the model repository
• 💬 Discussion: Use the Community tab on HuggingFace

---

Made with ❤️ for the Urdu-speaking community