Whisper-small fine-tuned for Marathi (FLEURS)

A fine-tune of openai/whisper-small for Marathi speech-to-text. This is a small learning project built in a few hours on a free Colab T4 GPU, not a production system.

Results (FLEURS Marathi test split, 1015 clips)

Notes on how to read these numbers:

• Both models were decoded the same way: greedy search, max 300 new tokens, fp16. The baseline was not given any protection against repetition loops, which inflates its WER (it can exceed 100%).
• Text was normalised before scoring (Unicode NFC, lowercase, punctuation and symbols removed), so these numbers are not directly comparable to leaderboard figures that use other normalisers.
• Word-level F1 and exact match are supplementary metrics. Speech recognition is not a classification task, so WER and CER are the main metrics.

How to use

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="joel-pawar/whisper-small-marathi-fleurs",
    chunk_length_s=30,
    generate_kwargs={"language": "marathi", "task": "transcribe"},
)
print(asr("your_audio.wav")["text"])

A smoke test is included: python test_model.py (or python test_model.py my_audio.wav).

Training

• Base model: openai/whisper-small
• Training data: FLEURS Marathi (mr_in) train split, through a parquet copy on Hugging Face (Srijan-Upadhyay/marathi-tts-fleurs). About 12 hours of clean read speech, with clips whose label sequence was too long for Whisper filtered out.
• Hyperparameters: learning rate 1e-5, effective batch size 16 (8 x 2 gradient accumulation), 50 warmup steps, 800 steps, fp16, gradient checkpointing
• Hardware: Tesla T4
• Library versions: transformers 5.19.0, torch 2.11.0+cu130

All raw results are in results/ (per-clip predictions, the 30 worst errors, training log, metrics.json).

Limitations

• Trained and tested only on FLEURS read speech: clean audio, one speaker at a time, a limited vocabulary and topics. Accuracy on phone calls, noisy audio, dialects, conversations or overlapping speakers is untested and probably much worse.
• A small amount of training data (about 12 hours) and a short training run. Expect errors on rare words, names and numbers.
• Output is Devanagari text without punctuation conventions.
• Not evaluated for fairness across speakers, gender or dialect.
• Not suitable for high-stakes uses such as legal, medical or safety-critical transcription.

License and citations

Released under the MIT license, the same as the base Whisper model. FLEURS is licensed CC BY 4.0.

• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022.
• Conneau et al., FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, 2022.