whisper-small-sinhala-run12
Full fine-tune of openai/whisper-small (244M parameters) for Sinhala automatic speech recognition. Reaches 15.52% WER / 4.36% CER on a speaker-disjoint test set of 15,860 utterances — the best result in the SinhaSpeech project.
It is the ASR model behind SinhaSpeech, a Sinhala accessibility platform built as a University of Moratuwa DSE project. The platform covers live lecture captioning, transcript review, spoken quizzes and a hands-free voice-command mode.
Usage
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="SinhaSpeech/whisper-small-sinhala-run12-best",
chunk_length_s=30,
)
print(asr("sample.wav", generate_kwargs={"language": "sinhala", "task": "transcribe"})["text"])
- Audio should be 16 kHz mono. The pipeline resamples common formats through ffmpeg.
- Whisper handles 30 s windows. Use
chunk_length_s=30, or window the audio yourself for longer recordings. - For low-latency use, the project serves a CTranslate2 (
faster-whisper) conversion of this model for live captioning.
Training
| Base model | openai/whisper-small |
| Method | Full fine-tune (every weight updated) |
| Data | stratified_v6 — SinhaSpeech/sinhala-asr-data-v6 |
| Split sizes | 123,205 train / 15,763 validation / 15,860 test, speaker-disjoint |
| Learning rate | 8.56e-5, cosine decay, 500 warmup steps |
| Batch size | 64 effective (64 per device × 1 accumulation step) |
| Epochs / steps | 6 epochs, 11,556 steps |
| Precision / hardware | bf16 on AMD Instinct MI300X (ROCm) |
| Augmentation | SpecAugment + noise/stretch/pitch, prob=0.2 each |
| Hyperparameter search | Optuna TPE sweep (20 trials); this is the best trial |
| Train runtime | ~5.75 h (final train loss 0.0238, average train loss ~0.089) |
| Final checkpoint | step 11,556 (epoch 6) |
Results
| Split | WER | CER |
|---|---|---|
| Validation (epoch 6) | 18.70% | 6.07% |
| Test (15,860 rows) | 15.52% | 4.36% |
Training curves
Train loss falls from ~2.41 to 0.024. Eval loss falls steadily from 0.169 to 0.076 by epoch 5, then ticks up slightly to 0.079 at epoch 6 while WER still improves marginally. This is a mild divergence — not clear overfitting.
| Epoch | Val WER | Val CER | Val loss |
|---|---|---|---|
| 1 | 31.18% | 9.92% | 0.1688 |
| 2 | 26.12% | 8.21% | 0.1087 |
| 3 | 22.52% | 7.02% | 0.1051 |
| 4 | 20.22% | 6.48% | 0.0858 |
| 5 | 18.75% | 6.06% | 0.0756 |
| 6 | 18.70% | 6.07% | 0.0787 |
The cosine schedule peaks at ~8.56e-5 after the 500-step warmup, then decays smoothly to near zero.
Comparison with other runs
| Run | Setup | Test WER | Test CER |
|---|---|---|---|
| run10 | v6 data, linear lr=3e-5, 5 ep | 17.17% | 4.71% |
| run11 | v6 data, linear lr=3e-5, 6 ep | 16.38% | 4.43% |
| run12 (this) | v6 data, cosine lr=8.56e-5, 6 ep | 15.52% | 4.36% |
The switch from linear to cosine schedule and the higher learning rate (found by the Optuna sweep) are what produced the improvement. At the same epoch count, run12 beats run11 by 0.86 points WER and 0.07 points CER.
Repository contents
The repo root holds the final model (model.safetensors, tokenizer and config files) and the training-curve images in images/.
Citation
@misc{jayasinghe2026whispersinhala,
author = {Yohan Jayasinghe},
title = {whisper-small-sinhala-run12: Whisper-small fine-tuned for Sinhala ASR},
year = {2026},
url = {https://huggingface.co/SinhaSpeech/whisper-small-sinhala-run12-best}
}
Built on Whisper (Radford et al., 2022). Training data includes OpenSLR-52 (Sinhala).