Whisper-small LoRA for Indian-English & Expressive Speech

A LoRA adapter for openai/whisper-small that makes it more robust on Indian-accented English, especially speech that is emotional or prosodically varied or that contains domain-specific proper nouns. Stock Whisper-small tends to struggle on this kind of audio.

	WER ↓
`openai/whisper-small` (base)	9.16%
+ this LoRA adapter	4.51% (−51% relative)

The adapter is only ~42 MB. It adds no inference latency once merged into the base model.

Model details


Base model	`openai/whisper-small` (244M params, 80 mel bins)
Method	LoRA (PEFT 0.18.1), `task_type=SEQ_2_SEQ_LM`
LoRA rank / alpha	r = 64, α = 128
LoRA dropout	0.05
Target modules	`q_proj`, `k_proj`, `v_proj` (encoder + decoder attention)
Language / task	English, `transcribe`
Audio	16 kHz mono, ≤ 30 s chunks
Developed by	Akash Sakala, built during an AI Research internship at Voibe (voice-AI)

Training data

A custom corpus of ~6,000 English clips built to cover the cases where Whisper-small fails most often on Indian speakers:

Accent: Indian-English speech
Emotion & prosody: emotionally expressive, varied-prosody delivery
Domain proper nouns: names and terms that the base model tends to garble

Evaluation

WER was measured on a held-out split of the same corpus.

Model	WER
whisper-small (zero-shot)	9.16%
whisper-small + LoRA (this)	4.51%

Usage

import torch, librosa
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

processor = WhisperProcessor.from_pretrained("openai/whisper-small")
base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(base, "Akash-Sakala/whisper-small-emotion")
model = model.merge_and_unload().eval()          # fold LoRA into the base weights

audio, _ = librosa.load("sample.wav", sr=16_000)  # 16 kHz mono
features = processor(audio, sampling_rate=16_000, return_tensors="pt").input_features

with torch.no_grad():
    ids = model.generate(features, language="en", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Limitations

Trained for English only. Code-mixed speech such as Hinglish is outside the training distribution.
The corpus is small (~6k clips), so gains may not carry over to very different domains or recording conditions such as telephony or heavy noise.
The WER comes from a held-out split of the same corpus. It is not a public benchmark.

Akash-Sakala/whisper-large-v3-svarah-indic: a QLoRA adapter for Whisper-large-v3 trained on Svarah, an Indian-accented English dataset
Akash-Sakala/qwen3.5-4b-transcript-formatter-lora: a post-processor that turns raw ASR output into clean, formatted text

Akash-Sakala/whisper-small-emotion