whisper-small-hi

This is a fine-tuned version of openai/whisper-small for automatic speech recognition (ASR) in Hindi. It was trained on the Hindi subset of Common Voice 17.0.

Model description

This model takes raw Hindi speech audio as input and outputs a text transcription. It is based on OpenAI's Whisper architecture (an encoder-decoder Transformer), starting from the pretrained whisper-small checkpoint and fine-tuned specifically to improve performance on Hindi.

Intended uses & limitations

Intended uses:

• Transcribing clear, single-speaker Hindi speech (e.g., voice notes, narration, read speech)
• As a starting point for further fine-tuning on a more specific Hindi speech domain (e.g., a particular accent, dialect, or noisy environment)

Limitations:

• Word Error Rate (WER) on the Common Voice test set is ~32%, meaning roughly 1 in 3 words may contain an error. This model is usable for casual or draft transcription but is not suitable for production, legal, medical, or other high-accuracy use cases without further fine-tuning or human review.
• Trained only on Common Voice's crowd-sourced read speech; performance may degrade on spontaneous conversational speech, heavy background noise, overlapping speakers, or strong regional accents not well represented in the training data.
• Only tested on 16kHz mono audio, consistent with Whisper's expected input format.

How to use

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = WhisperProcessor.from_pretrained("ranjan-prabhat/whisper-small-hi")
model = WhisperForConditionalGeneration.from_pretrained("ranjan-prabhat/whisper-small-hi").to(device)

# audio_array: a 1D numpy array of audio samples at 16kHz
input_features = processor(
    audio_array, sampling_rate=16000, return_tensors="pt"
).input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(input_features, language="hindi", task="transcribe")

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

If loading audio from a file, you can use librosa.load(path, sr=16000) or the Hugging Face datasets library's Audio feature (cast to a 16000 Hz sampling rate) to get the correctly formatted array.

Training and evaluation data

• Dataset: fsicoli/common_voice_17_0, Hindi (hi) configuration
• Train split: 4,689 examples
• Validation split: 2,428 examples
• Test split: 3,154 examples

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: AdamW (torch), betas=(0.9, 0.999), epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 2000
• mixed_precision_training: Native AMP (fp16)

Training results

The best checkpoint (by WER on the validation set) was automatically selected and used as the final model.

Framework versions

• Transformers 5.0.0
• PyTorch 2.4.1+cu121
• Datasets 3.6.0
• Tokenizers 0.22.2