Whisper Small — Santali ASR (Custom Transliteration / Sanlish)

Fine-tuned version of openai/whisper-small for Automatic Speech Recognition (ASR) of the Santali language, using a custom Roman-script transliteration scheme called Sanlish.

This model is part of a thesis research pipeline for low-resource Santali speech processing, targeting the task of Santali Speech → Sanlish Transcription → Bangla Translation.

---

Model Details

Model Description

• Developed by: thunderboltc
• Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
• Language: Santali (sat) — transcribed in custom Roman transliteration (Sanlish)
• Base model: openai/whisper-small
• Parameters: 241,734,912 (~242M)
• Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
• License: MIT

What is Sanlish?

Sanlish is a custom Roman-script transliteration of Santali designed to bridge the gap between Santali phonetics and Latin-alphabet-based ASR systems. It was developed as part of this thesis to serve as an intermediate representation between Santali audio and Bangla translation.

---

Training Details

Training Data

• Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
• Collected by: thunderboltc (thesis dataset, not yet publicly released)
• Split:

• Audio format: WAV, 16kHz mono
• Text target: Custom Roman transliteration (Sanlish)

Training Procedure

Preprocessing

• Audio resampled to 16,000 Hz
• Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
• Text tokenized using WhisperTokenizer with language=english, task=transcribe
• forced_decoder_ids and suppress_tokens disabled to allow custom vocabulary transcription

Training Hyperparameters

Best Checkpoint

Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).

---

Evaluation

Results on Test Set (51 utterances)

These results are consistent with expectations for a low-resource ASR system trained on ~400 utterances. The large gap between WER and CER reflects that the model captures phonetic character structure well, even when full word sequences deviate from the reference.

Sample Predictions

---

How to Get Started

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")

# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)

# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(inputs.input_features)

# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])

---

Intended Use

Direct Use

Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.

Downstream Use

This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.

Out-of-Scope Use

• Not suitable for general-purpose ASR in other languages
• Not recommended for standard Santali Ol Chiki script transcription
• Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set

---

Limitations & Bias

• Trained on only 502 utterances — a very small dataset by ASR standards
• May not generalize well to unseen speakers or recording environments
• Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
• No speaker diversity information available for the training set

---

Citation

If you use this model in your research, please cite:

@misc{thunderboltc2025whisper-santali,
  author = {thunderboltc},
  title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}

---

Environmental Impact

• Hardware: NVIDIA Tesla T4
• Cloud Provider: Google Colab
• Hours used: ~1.25 hours
• Carbon emissions: Estimated using the ML Impact Calculator