Whisper-large-v3 QLoRA for Indian-Accented English (Svarah)

A QLoRA adapter for openai/whisper-large-v3 fine-tuned on Svarah from AI4Bharat. Svarah has 9.6 hours of English speech from 117 speakers across 65 districts in 19 Indian states. The adapter adapts Whisper's largest model to Indian accents. The base model was loaded in 4-bit NF4, so it trains on a single mid-range GPU.

The adapter is only ~63 MB. Load it on top of the public base model; see Usage.

Model details


Base model	`openai/whisper-large-v3` (1.55B params, 128 mel bins)
Method	QLoRA: base model frozen in 4-bit NF4 with double quantisation (fp16 compute), LoRA adapters trained in fp16
LoRA rank / alpha	r = 32, α = 64
LoRA dropout	0.05
Target modules	`q_proj`, `v_proj`
Trainable params	~15.7M (~1% of base)
Language / task	English, `transcribe`
Developed by	Akash Sakala

Training

Hyperparameter	Value
Data	Svarah, ~90/10 train/eval split (~6,000 train clips)
Steps	500 (≈1.33 epochs)
Effective batch	16 (8 × grad-accum 2)
Learning rate	1e-3, linear decay, 50 warm-up steps
Optimizer	AdamW (fused)
Precision	fp16 mixed + gradient checkpointing
Best checkpoint	lowest eval loss (`load_best_model_at_end`)
Wall-clock	~94 min, single GPU

Loss curve

Step	Train loss	Eval loss
100	0.246	0.255
200	0.238	0.244
300	0.225	0.219
400	0.140	0.205
500	0.109	0.202

Eval loss fell 21% between steps 100 and 500 (0.255 → 0.202). After the first epoch (step 375), train loss drops sharply while eval loss flattens, so 500 steps is about where overfitting starts. The TensorBoard logs are in runs/.

Word error rate (WER) on the held-out split hasn't been measured yet. Eval loss is the only held-out metric reported here.

Usage

import torch, librosa
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
base = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "Akash-Sakala/whisper-large-v3-svarah-indic")
model = model.merge_and_unload().eval()

audio, _ = librosa.load("sample.wav", sr=16_000)
features = processor(audio, sampling_rate=16_000, return_tensors="pt").input_features
features = features.to(model.device, torch.float16)

with torch.no_grad():
    ids = model.generate(features, language="en", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Low-VRAM option: load the base model with BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4") as it was during training, and skip merge_and_unload().

Limitations

Svarah is published as an evaluation benchmark. Because this adapter trains on part of it, don't compare scores on Svarah with published zero-shot numbers. If the split was made per clip rather than per speaker, eval speakers may also appear in training.
English only. Hindi–English code-mixing is outside the training distribution.
Svarah is small (~9.6 h), so the adapter mainly shifts the model toward the accent. It doesn't teach new vocabulary at scale.

Akash-Sakala/whisper-small-emotion: a LoRA for Whisper-small on Indian-English expressive speech (WER 9.16% → 4.51%)
Akash-Sakala/qwen3.5-4b-transcript-formatter-lora: an ASR post-processor that formats raw transcripts

Citation (dataset)

@inproceedings{javed2023svarah,
  title     = {Svarah: Evaluating English ASR Systems on Indian Accents},
  author    = {Javed, Tahir and Joshi, Sakshi and Nagarajan, Vignesh and Sundaresan, Sai and Nawale, Janki and Raman, Abhigyan and Bhogale, Kaushal and Kumar, Pratyush and Khapra, Mitesh M.},
  journal   = {arXiv preprint arXiv:2305.15760},
  year      = {2023}
}

Akash-Sakala/whisper-large-v3-svarah-indic