🎙️ BaltiVoice ASR — Whisper Small Fine-Tuned for Balti (bft)

First public Automatic Speech Recognition model for Balti, a critically low-resource Tibetic language spoken in Gilgit-Baltistan, Pakistan.

📦 Dataset • 🎧 Live Demo • 💻 GitHub • 📄 Paper

---

Model Details

Model Description

This model is a fine-tuned version of openai/whisper-small for Automatic Speech Recognition (ASR) in the Balti language (bft).

Balti is a Tibetic language with roughly 400,000 speakers, written in Nastaliq (Arabic-based) script. Before this work, no publicly available ASR models or datasets existed for Balti. This model transcribes Balti speech into native Nastaliq text.

• Developed by: Muhammad Ali, Independent Researcher, Gilgit-Baltistan, Pakistan. Alumnus, The Islamia University of Bahawalpur (IUB).
• Model type: Sequence-to-sequence ASR (Whisper architecture)
• Language: Balti (bft)
• License: Apache 2.0
• Base model: openai/whisper-small

Model Sources

• Repository: github.com/mohdali-dev/BaltiVoice-ASR
• Demo: HuggingFace Spaces
• Paper: arXiv:2606.03504

---

Results

Zero-shot WER above 100% indicates hallucination — the model generates words not present in the reference. Fine-tuning on 16.8 hours of Balti speech reduces this to an impressive 26.74% WER and 8.67% CER on the 538-utterance speaker-disjoint validation set.

---

How to Get Started

Installation

pip install transformers torch librosa

Inference

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="mohdali1/whisper-small-balti",
    generate_kwargs={"language": "urdu", "task": "transcribe"}
)

result = asr("your_balti_audio.wav")
print(result["text"])

Manual inference

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch
import librosa

model_id  = "mohdali1/whisper-small-balti"
processor = WhisperProcessor.from_pretrained(
    model_id, language="urdu", task="transcribe"
)
model     = WhisperForConditionalGeneration.from_pretrained(model_id)

audio, sr = librosa.load("your_balti_audio.wav", sr=16000)
inputs    = processor(audio, sampling_rate=16000, return_tensors="pt")

with torch.no_grad():
    generated_ids = model.generate(inputs.input_features)

transcription = processor.batch_decode(
    generated_ids, skip_special_tokens=True
)[0]
print(transcription)

---

Uses

Direct Use

• Transcription: Convert Balti audio into native Nastaliq text
• Research: Study low-resource ASR and transfer learning for Tibetic languages
• Education: Build tools for Balti literacy and pronunciation

Downstream Use

• Voice assistants for Balti speakers
• Media archiving of radio broadcasts, folk stories, oral histories
• Healthcare documentation in rural Gilgit-Baltistan settings

Out-of-Scope Use

• High-stakes decisions (legal, medical, safety-critical) without human verification — WER is ~27%, not production-ready
• Other languages — performance on non-Balti input is not guaranteed
• Commercial deployment without further domain-specific evaluation

---

Training Details

Training Data

• Dataset: BaltiVoice ASR Dataset
• Total clips: 10,060 validated utterances (~16.8 hours)
• Format: 16kHz mono WAV, native Nastaliq transcriptions
• Split method: Speaker-disjoint (GroupShuffleSplit on client_id, seed 42)

Training Hyperparameters

Training Curve

Note: The raw training WER at step 1,000 was 30.07%. However, the final normalized evaluation (with punctuation removed) on the speaker-disjoint held-out set yielded the reported 26.74% WER and 8.67% CER, confirming the model generalizes well to unseen speakers.

---

Bias, Risks, and Limitations

Technical Limitations

• WER of 26.74% — roughly one word in four may be incorrect. Not suitable for critical applications without human review.
• Read speech only — trained on short read clips (avg 6 seconds). Performance on spontaneous conversational speech will likely be lower.
• No Unicode normalization — Nastaliq script Unicode ambiguities (e.g., Arabic Yeh vs. Farsi Yeh) may affect output consistency.
• Speaker diversity — 136 speakers, mostly from Gilgit-Baltistan. Dialectal variation from other regions may affect accuracy.

Sociotechnical Considerations

• Balti is an endangered language. Mis-transcriptions could distort cultural meaning. Native speaker validation is recommended.
• The dataset represents a specific regional subset of Balti speakers and may not capture all dialectal variation.

Recommendations

• Use human review for sensitive or important content
• Encourage Balti speakers to report errors via GitHub Issues
• Consider extended training or Whisper-medium for higher accuracy

---

Environmental Impact

Estimated using the ML Impact Calculator (Lacoste et al., 2019).

• Hardware: NVIDIA Tesla T4
• Training time: ~1.9 hours
• Cloud provider: Google Colab
• Carbon emitted: ~0.1 kg CO₂eq (estimated)

---

Citation

If you use this model or the associated dataset in your research, please cite:

@misc{ali2026baltivoice,
  author    = {Muhammad Ali},
  title     = {BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language},
  year      = {2026},
  eprint    = {2606.03504},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url       = {https://arxiv.org/abs/2606.03504}
}

---

Glossary

• WER: Word Error Rate = (Substitutions + Deletions + Insertions) / Total Words. Lower is better.
• CER: Character Error Rate. Useful for Nastaliq script where Unicode ambiguities can inflate WER.
• Nastaliq: Arabic-based script used for Urdu, Persian, and Balti.
• Low-resource language: A language with limited digital data, tools, and models available for NLP/ASR.
• Speaker-disjoint split: Train and validation sets contain entirely different speakers, preventing the model from memorizing speaker acoustics.

---

More Information

• Dataset: mohdali1/baltivoice-asr
• Demo: baltivoice-demo
• GitHub: BaltiVoice-ASR
• Paper: arXiv:2606.03504
• Author: Muhammad Ali
• Contact: s22bseen1m01052@iub.edu.pk | ORCID | LinkedIn