Beyond Words — Whisper-medium Bangla dialect classifier

Encoder-only WhisperForAudioClassification fine-tuned to identify nine Bangla dialect classes from 5-second clips: Barishal, Chattogram, Dhaka, Khulna, Mymensingh, Noakhali, Rajshahi, Standard (the standard register), Sylhet. The checkpoint's id2label keeps the corpus label Formal for the Standard class.

Trained for the undergraduate thesis A Neural Network Approach to Classifying Bangla Accentual Diversity: Dialect Identification from Sound and from Words (East Delta University, 2026). Code: https://github.com/towfique-elahe/beyond-words. Live demo: https://beyondwords.towfiqueelahe.com.

Results (test macro-F1)

Per-class F1 on the strict test split: Barishal 0.88, Chattogram 0.85, Dhaka 0.97, Khulna 0.97, Mymensingh 0.91, Noakhali 0.90, Rajshahi 0.95, Standard 0.91, Sylhet 0.94.

How it was trained

• Base: openai/whisper-medium encoder; decoder dropped; projector → mean-pool → linear head.
• Inputs: first 5 s of 16 kHz audio as 80-band log-mel (500 frames). The positional table is cut to 250 rows, so this model accepts 5 s inputs only (max_source_positions = 250); longer audio should be scored in 5 s windows.
• Data: 9,303 clips (13.1 h) of Bangladeshi TV, film and online speech (Badhon et al., 2020; Alam et al., 2023), strict pseudo-speaker split (ECAPA-TDNN clustering): 6,931 / 1,045 / 1,327 clips.
• Recipe: class-weighted cross-entropy, encoder lr 2e-5, head lr 1e-4, batch 8 × 2, fp16, 12 epochs on a Kaggle T4; best validation macro-F1 at epoch 5.

Usage

import numpy as np, torch, librosa
from transformers import WhisperFeatureExtractor, WhisperForAudioClassification

repo = "towfique-elahe/beyond-words-whisper-medium-did"
fe = WhisperFeatureExtractor.from_pretrained(repo)
model = WhisperForAudioClassification.from_pretrained(repo).eval()

wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
wav = np.pad(wav[:80000], (0, max(0, 80000 - len(wav))))          # first 5 s
x = fe(wav, sampling_rate=16000, padding="max_length", max_length=80000, return_tensors="pt").input_features
probs = torch.softmax(model(input_features=x).logits, -1)[0]
print(model.config.id2label[int(probs.argmax())], float(probs.max()))

Limitations

Trained on broadcast and online media only; telephone or noisy speech and speakers from regions outside the nine classes are mapped to the nearest known class. Single training seed. Class boundaries partly reflect register (Standard) rather than region.