Whisper Large-v3-Turbo Hindi LoRA

A LoRA fine-tuned adapter for openai/whisper-large-v3-turbo optimized for Hindi (Devanagari) speech recognition.

Results

37.4% relative WER reduction. INT8 deployment via faster-whisper adds only 0.45% WER degradation.

Evaluation uses Whisper-default text normalization. See Normalization Notes below.

Comparison with Other Hindi ASR Models

Note: The collabora and vasista22 models are full fine-tunes trained on hundreds of hours of multi-corpus Hindi data. This model uses only ~3.5 hours of FLEURS data with a lightweight LoRA adapter, making it a fundamentally different trade-off: minimal data and compute for significant WER improvement over the zero-shot baseline.

Training Curve

Best checkpoint: step 300 (lowest val WER). Test WER: 22.25%.

How to Use

With PEFT (LoRA adapter)

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel

BASE_MODEL = "openai/whisper-large-v3-turbo"
ADAPTER = "Tachyeon/whisper-large-v3-turbo-hindi-lora"

processor = WhisperProcessor.from_pretrained(BASE_MODEL)
base_model = WhisperForConditionalGeneration.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model = model.to("cuda").eval()

# Transcribe (audio_array: 16kHz float32 numpy array)
input_features = processor(
    audio_array, sampling_rate=16000, return_tensors="pt"
).input_features.to("cuda", dtype=torch.bfloat16)

with torch.inference_mode():
    predicted_ids = model.generate(
        input_features, language="hi", task="transcribe"
    )

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

With faster-whisper (merged + CTranslate2)

For production deployment, merge the adapter and convert to CTranslate2:

# Merge LoRA → convert → evaluate
python convert_and_eval.py --lora-dir outputs/whisper-large-v3-turbo-hindi-lora --quant int8 --gpu 0

from faster_whisper import WhisperModel

model = WhisperModel("path/to/ct2-model", device="cuda", compute_type="int8")
segments, info = model.transcribe("audio.wav", language="hi", beam_size=1)
print(" ".join(seg.text.strip() for seg in segments))

Full pipeline code (data prep → training → deployment): github.com/ipritamdash/whisper-hindi-lora

LoRA Configuration

Architecture choice follows LoRA-Whisper (arXiv:2406.06619): encoder+decoder targeting on all linear layers outperforms decoder-only or q/v-only configurations.

Training Details

Framework Versions

• Transformers: 4.57.3
• PEFT: 0.18.1
• PyTorch: 2.6.0+cu124
• Datasets: 3.6.0

Dataset

Google FLEURS Hindi (hi_in):

• Domain: Read speech from Wikipedia sentences
• Audio: 16kHz mono, Devanagari script
• License: CC BY 4.0
• Size: ~3.5 hours across train/val/test

Normalization Notes

Hindi ASR evaluation is sensitive to text normalization. Whisper's default normalizer strips diacritics and simplifies conjunct consonants, which can inflate apparent accuracy but loses semantic precision.

WER numbers above use Whisper-default normalization for comparability with other HuggingFace models. For production Hindi ASR, consider evaluation with IndicNLP normalizer.

Limitations

• Training data scope: Trained on FLEURS read speech (~3.5h). Performance on conversational, noisy, or accented Hindi may vary.
• Language detection: Fine-tuning on a single language can degrade Whisper's multilingual detection. Set language="hi" explicitly.
• Code-mixing: Performance on Hindi-English (Hinglish) is not evaluated.
• Base model biases: Any biases in whisper-large-v3-turbo carry through.

Citation

@misc{dash2026whisper_hindi_lora,
  author = {Pritam Dash},
  title = {Whisper Large-v3-Turbo Hindi LoRA Fine-tune},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Tachyeon/whisper-large-v3-turbo-hindi-lora}
}

References

• Whisper paper (Radford et al., 2023)
• LoRA paper (Hu et al., 2021)
• LoRA-Whisper (Yang et al., 2024)
• FLEURS (Conneau et al., 2023)