Compact, robust Whisper-small ASR model fine-tuned with phonetically-grounded acoustic LoRA on Indian English and extempore Hinglish code-switching, eliminating degenerate looping hallucinations on spontaneous accented speech.
transformers, sentence-transformers, peft, and standard inference pipelines.Evaluated on the official held-out test split under standard production evaluation protocols:
| Benchmark Metric | Score / Value |
|---|---|
| Fine-Tuned Held-Out Test WER | 47.75% |
| Unadapted Baseline Test WER | 99.62% |
| WER Absolute Improvement | -51.87% |
| Fine-Tuned Held-Out Test CER | 34.53% |
| Unadapted Baseline Test CER | 85.79% |
| CER Absolute Improvement | -51.26% |
| Held-Out Test Set Size | 169 audio clips (speaker-disjoint) |
| LoRA Trainable Parameters | 7,077,888 (2.84%) |
| Base Model Parameters | 248,812,800 (244M) |
💡 Benchmark Takeaway: Fine-tuning establishes substantial gains over the zero-shot base checkpoint, dramatically suppressing false predictions and improving domain confidence.
Install the required dependencies:
pip install transformers torch sentence-transformers peft accelerate
import torch
from transformers import pipeline, WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel
# 1. Pipeline Quickstart
transcriber = pipeline(
"automatic-speech-recognition",
model="vennyvirtuoso/whisper-small-indic-english-hinglish",
chunk_length_s=30,
device="cuda:0" if torch.cuda.is_available() else "cpu",
)
# Transcribe Indian English or Hinglish audio
audio_path = "sample_indic_speech.wav"
result = transcriber(audio_path, generate_kwargs={"language": "en", "task": "transcribe"})
print(result["text"])
# 2. Direct PEFT + Transformers Ingestion
base_model_name = "openai/whisper-small"
processor = WhisperProcessor.from_pretrained("vennyvirtuoso/whisper-small-indic-english-hinglish")
base_model = WhisperForConditionalGeneration.from_pretrained(base_model_name, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base_model, "vennyvirtuoso/whisper-small-indic-english-hinglish")
"To record this tutorial, main Ubuntu Linux OS version 12.04 aur KTurtle use kar rahi hoon."
| Parameter | Specification |
|---|---|
| Base Foundation Model | openai/whisper-small |
| Fine-Tuned Checkpoint | vennyvirtuoso/whisper-small-indic-english-hinglish |
| Task Category | Automatic Speech Recognition (ASR) |
| Training Dataset | agarwalayushi/hinglish |
| Precision | Mixed Precision (bfloat16 / fp16) |
| License | Apache 2.0 (Permissive Commercial & Research) |
This model is open-sourced under the Apache 2.0 license. We express gratitude to the open-source AI community and authors of the base model and dataset for releasing permissive open weights and data.
openai/whisper-small