Whisper Large V3 — Luganda NSS v24 (Encoder + SpecAugment + Data Augmentation)

Stage 2 fine-tune on the CDLI Ugandan Luganda Nonstandard Speech dataset. Built on top of lg_waxal_tune_whisper_large_7 (Stage 1 with SpecAugment). This is Variant 3 of the Stage 2 experiments — encoder only with SpecAugment and data augmentation.

Training Details

Dataset: cdli/ugandan_luganda_nonstandard_speech_v1.0
Stage 1 checkpoint: lg_waxal_tune_whisper_large_7/best_model
Encoder frozen: No (encoder trained only)
Decoder frozen: Yes
Projection frozen: Yes
SpecAugment: On
Data augmentation: Gaussian noise (p=0.5), gain perturbation (p=0.5), speed perturbation via torchaudio resample (p=0.5)
Learning rate: 5e-5 (polynomial decay)
Max steps: 1000
Best checkpoint: Step 300
Output dir: lg_nss_tune_whisper_large_v24

Results (checkpoint-300)

Metric	Value
Overall WER (normalized)	0.76
Overall CER (normalized)	0.297
Avg WER (normalized)	0.544
Avg CER (normalized)	0.181
Meaning preservation	82.6%

WER by Severity

Severity	Baseline WER	Fine-tuned WER
Mild	0.89	0.49
Moderate	0.96	0.55
Severe	0.97	0.64

WER by Etiology

Etiology	Baseline WER	Fine-tuned WER
Acquired hearing impairment	0.98	0.71
Cerebral Palsy	0.95	0.45
Cerebral palsy - cerebral malaria	0.93	0.57
Developmental	0.97	0.56
Idiopathic	0.80	0.40
Structural (tongue tie)	0.96	0.66

WER by Speaker

Speaker	Severity	Etiology	Baseline WER	Fine-tuned WER
UG001	Mild	Cerebral palsy - cerebral malaria	0.93	0.57
UG014	Mild	Idiopathic	0.80	0.40
UG022	Mild	Developmental	0.95	0.51
UG021	Moderate	Structural (tongue tie)	0.96	0.66
UG036	Moderate	Cerebral Palsy	0.95	0.45
UG052	Moderate	Developmental	0.98	0.55
UG042	Severe	Developmental	0.98	0.61
UG057	Severe	Acquired hearing impairment	0.98	0.71
UG058	Severe	Developmental	0.98	0.59

Comparison with Other Variants (Stage 2)

Variant	Config	Avg WER	Meaning Preservation
v22 (Run 4)	Full model + SpecAugment	0.534	84.7%
v23 (Run 5)	Encoder only + SpecAugment	0.555	80.4%
v24 (Run 6)	Encoder only + SpecAugment + Data Aug	0.544	82.6%

ElizabethMwangi/whisper-large-v3-luganda-nss-v24