Whisper Small.en — Fine-tuned
Fine-tuned openai/whisper-small.en using LoRA (PEFT) on English speech data from Google FLEURS (en_us).
Built for the s2st-en-hi real-time speech-to-speech translation pipeline (English ↔ Indic languages).
Training details
| Base model | openai/whisper-small.en (244M params) |
| Method | LoRA (r=16, alpha=32) — 5.6M trainable / 247M total (2.3%) |
| Data | 1,800 train / 200 val clips from FLEURS en_us |
| Epochs | 3 |
| Batch size | 2 x 8 gradient accumulation = 16 effective |
| Learning rate | 1e-4 with warmup (50 steps) |
| Device | Apple M4 (MPS), 29 min total |
| Peak RAM | ~5.2 GB |
Results
| Epoch | WER | Eval Loss |
|---|---|---|
| 1 | 7.2% | 0.214 |
| 2 | 5.9% | 0.192 |
| 3 | 5.76% | 0.192 |
A/B test on 15 TTS-synthesized English phrases:
- Baseline
small.en: 7/15 (47%) exact match - Fine-tuned: 12/15 (80%) exact match
- Latency: identical (~1.4s/utterance on CPU int8)
Usage
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model = WhisperForConditionalGeneration.from_pretrained("ronxldwilson/whisper-small-en-finetuned")
processor = WhisperProcessor.from_pretrained("ronxldwilson/whisper-small-en-finetuned")
For the CTranslate2 int8 version (production, faster-whisper): see ronxldwilson/whisper-small-en-finetuned-ct2-int8