Whisper-base Hinglish (romanized)

A fine-tune of openai/whisper-base that transcribes Hindi, English and code-switched Hinglish speech from India directly into Roman script, the way people type it (kya scene hai, not क्या सीन है). English stays verbatim and Hindi comes out in everyday Roman spelling.

• 74 M parameters, 16 kHz, max 30 s per window (longer audio is windowed automatically).
• Output is always Roman, with no separate transliteration step at inference.
• Built for on-device dictation; it is the speech model of Yapper.

Files

The int8 build is within 0.2 WER points of fp32 on the held-out test sets below (same decoding, GPU).

Usage

Transformers

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="abhishekgautamm/whisper-base-hinglish", chunk_length_s=30)
print(asr("clip.wav", generate_kwargs={"language": "en", "task": "transcribe"})["text"])

faster-whisper (CTranslate2)

from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel

path = snapshot_download("abhishekgautamm/whisper-base-hinglish-v2", allow_patterns=["ct2/int8/*"])
model = WhisperModel(f"{path}/ct2/int8", device="cpu", compute_type="int8")
segments, _ = model.transcribe("clip.wav", language="en", beam_size=1, without_timestamps=True)
print("".join(s.text for s in segments).strip())

Always decode with language="en" and without timestamps (without_timestamps=True in faster-whisper; transformers does this by default). The model was trained to emit Roman text under the English token, for Hindi speech too. Adding no_repeat_ngram_size=3, repetition_penalty=1.1 (as in the evaluation) suppresses rare repetition loops.

Evaluation

Word / character error rate (%), greedy decoding, fp32. References and outputs are lower-cased, punctuation-stripped and digits spelled out; Devanagari references are romanized with LIPI (the same tool used to build the training targets).

Public benchmarks, full test sets (none of these were used in training)

Held-out test sets of the training corpora (official test splits; SPIRE/wSPIRE by held-out speaker, VoiceBench by held-out prompt)

Training

About 250 hours, with every target romanized by LIPI: MUCS 2021 Hindi-English (86 h), Common Voice English (48 h), IISc wSPIRE (31 h), IISc SPIRE-SIES (26 h), MUCS 2021 Hindi (14 h), VoiceBench (13 h), SLURP (7 h), Timers and Such (5 h), HiACC (4 h), Fluent Speech Commands (0.5 h). The language token is forced to en. Training ran for 3 epochs (released checkpoint), LR 1e-4 with 300 warm-up steps, fp16 mixed precision.

Limitations

• Roman spelling of Hindi is phonetic and can vary (hai / hain, ki / kii).
• Little punctuation or casing.
• Weakest on long-form conversational Hinglish (CoSHE) and read Hindi; best on short dictation-style speech.
• Hindi, English and Hinglish only; other Indic languages are out of scope.
• The Hindi scores use the same romanizer as the training targets, which favours this model's spelling conventions.
• The training corpora carry their own licenses; check them before commercial use.