Whisper Small — Bengali ASR (fine-tuned)

This model is a fine-tuned version of Noobbbbb/whisper-small-bn on the Noobbbbb/bengali-ai-asr-80k dataset, for automatic speech recognition (ASR) in Bengali.

Model Details

• Base model: Noobbbbb/whisper-small-bn (Whisper-small architecture, previously fine-tuned for Bengali)
• Architecture: Whisper small (12 encoder layers, 12 decoder layers, 768 hidden size)
• Language: Bengali (bn)
• Task: transcribe
• Fine-tuning framework: 🤗 transformers Seq2SeqTrainer

Training Data

• Dataset: Noobbbbb/bengali-ai-asr-80k
• Train subset used: first 40,000 examples of the train split
• Validation subset used: first 4,000 examples of the val split
• Audio was processed via WhisperProcessor (log-mel spectrogram input features); transcripts were tokenized with the Whisper tokenizer.

Training Procedure

Training Results — Stage 1 (initial fine-tune)

• Final training loss: 0.2969
• Total steps: 1,875 (3 epochs)
• Train runtime: ~6,500s (~108 minutes)
• Throughput: ~18.5 samples/sec

Evaluation used Word Error Rate (WER) and Character Error Rate (CER) via the evaluate library (wer_metric / cer_metric), computed every 400 steps on the 4,000-example validation subset.

Stage 1 final validation WER: 40.75% · CER: 12.62%

Training Results — Stage 2 (extended fine-tune, +5 epochs)

Training was continued for 5 additional epochs from the Stage 1 checkpoint.

• Final training loss: 0.1609
• Total steps: 6,645 (5 epochs this stage)
• Train runtime: ~17,215s (~4.78 hours)
• Throughput: ~24.7 samples/sec

Stage 2 out-of-domain evaluation:

Evaluated on google/fleurs (bn_in config)

This was run on an out-of-domain evaluation set (i.e. not the in-domain validation subset used for Stage 1 tracking). Compared to Stage 1's in-domain validation numbers, WER improved (40.75% → 38.90%) while CER is slightly higher (12.62% → 13.98%) — consistent with evaluating on different, harder data rather than a straightforward regression.

Sample Predictions (Stage 2)

Qualitatively, Stage 2 predictions track the reference more closely than Stage 1 (fewer inserted/garbled syllables), though small substitution and diacritic errors remain.

How to Use

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch, librosa

repo_id = "Noobbbbb/bengali-ai-asr-80k"  # update if you rename the repo

processor = WhisperProcessor.from_pretrained(repo_id)
model = WhisperForConditionalGeneration.from_pretrained(repo_id)

model.generation_config.language = "bengali"
model.generation_config.task = "transcribe"

waveform, sr = librosa.load("audio.wav", sr=16000, mono=True)
inputs = processor(waveform, sampling_rate=16000, return_tensors="pt")

with torch.no_grad():
    predicted_ids = model.generate(inputs["input_features"])

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

For longer audio

from transformers import pipeline, WhisperForConditionalGeneration, WhisperProcessor, AutoModelForSpeechSeq2Seq

model_id = "Noobbbbb/whisper-small-bn-v2"

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)

if isinstance(model.generation_config.eos_token_id, list):
    model.generation_config.eos_token_id = model.generation_config.eos_token_id[0]

if isinstance(model.config.eos_token_id, list):
    model.config.eos_token_id = model.config.eos_token_id[0]

processor = WhisperProcessor.from_pretrained(model_id)

waveform, _ = librosa.load("/content/yt.mp3")

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=0 if torch.cuda.is_available() else -1,
)

result = pipe(
    waveform,
    return_timestamps=True,   # enables long-form transcription
    generate_kwargs={"language": "bengali"}
)

Note: This will return timestamp-wise transcription

Limitations

• Stage 1 was fine-tuned only on a 40k-example subset of the full dataset; Stage 2's exact train subset size is unconfirmed (see note above).
• Stage 1 metrics (WER 40.75%, CER 12.62%) are in-domain validation numbers; Stage 2 metrics (WER 38.90%, CER 13.98%) are from an out-of-domain evaluation set, so the two aren't directly comparable apples-to-apples.
• Performance on noisy/telephony-style audio (e.g. call center recordings) has not been separately validated for this checkpoint.

Citation / Acknowledgements

• Base model: Noobbbbb/whisper-small-bn
• Dataset: Noobbbbb/bengali-ai-asr-80k
• Built on OpenAI's Whisper architecture via 🤗 Transformers.