whisper-small-sinhala-run15-backup
Fine-tuned openai/whisper-small for Sinhala (si) ASR with mixed-data anti-catastrophic-forgetting training.
This is the new project best model, beating run12 on the Sinhala test set while dramatically reducing English catastrophic forgetting.
Results
Sinhala ASR (stratified_v6 test set, 15,860 samples)
| Metric | Value |
|---|---|
| Test WER | 15.48% |
| Test CER | 4.24% |
| Val WER (epoch 6) | 18.60% |
| Val CER (epoch 6) | 5.87% |
English Catastrophic Forgetting (LibriSpeech test-clean, 2,620 samples)
| Model | English WER | Delta vs base | Severity |
|---|---|---|---|
| whisper-small base | 4.27% | — | — |
| run12 (pure Sinhala) | 98.70% | +94.43 pts | severe |
| run15 (this model) | 8.00% | +3.73 pts | mild |
The 7.5% English mix in training reduced catastrophic forgetting from +94 pts to +3.73 pts — a 96% reduction.
Training Details
| Parameter | Value |
|---|---|
| Base model | openai/whisper-small |
| Dataset | stratified_v7 (mixed) |
| Train set | 133,205 rows — 92.5% Sinhala + 7.5% English |
| Val/Test set | Sinhala-only (v6 split, comparable across runs) |
| Learning rate | 8.56e-5 |
| LR scheduler | cosine |
| Batch size | 64 |
| Epochs | 6 |
| Warmup steps | 500 |
| Noise aug prob | 0.2 |
| Stretch aug prob | 0.2 |
| Pitch aug prob | 0.2 |
| Total steps | 12,492 |
Validation WER per Epoch
| Epoch | Val WER | Val CER |
|---|---|---|
| 1 | 31.64% | 9.78% |
| 2 | 25.62% | 7.95% |
| 3 | 22.42% | 6.90% |
| 4 | 20.41% | 6.32% |
| 5 | 18.75% | 5.91% |
| 6 | 18.60% | 5.87% |
Weight Delta Analysis (run15 vs run12)
Per-layer L2 norm of (run15 − run12) relative to run12 base norm, showing which parts of the model changed most due to the English data mix.
Most Changed Layers
| Layer | Relative Change |
|---|---|
| decoder.embed_positions | 60.0% |
| decoder.embed_tokens | 46.1% |
| decoder.layers.*.encoder_attn.out_proj | 39–49% |
| decoder.layers.*.encoder_attn.{k,v}_proj | 40–45% |
| decoder.layers.*.self_attn.out_proj | 39–45% |
Least Changed Layers
| Layer | Relative Change |
|---|---|
| encoder.layers.*.final_layer_norm.{weight,bias} | 0.09–0.16% |
Key insight: The decoder changed dramatically (~40–60% in cross-attention and embeddings) while the encoder barely moved (~0.1% in layer norms). The English mix kept the decoder's cross-attention and positional embeddings anchored to English representations, while the encoder learned Sinhala acoustics similarly to run12. This explains why English forgetting dropped so dramatically — the decoder retained its English language modelling capability.
Anti-Forgetting Strategy
Previous runs (run12, run14) used pure Sinhala training data and showed severe catastrophic forgetting (+94–96 pts English WER delta). This run adds 10,000 LibriSpeech train-clean-100 English samples (7.5% of training data) to the v6 Sinhala training set. Each row carries a language column ("si" / "en") and is tokenized with the correct language prefix token (<|si|> / <|en|>) during training.
Usage
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch
model = WhisperForConditionalGeneration.from_pretrained("Yohan2003/whisper-small-sinhala-run15-backup")
processor = WhisperProcessor.from_pretrained("Yohan2003/whisper-small-sinhala-run15-backup")
# For Sinhala transcription
model.generation_config.language = "sinhalese"
model.generation_config.task = "transcribe"