whisper-small-mn-v25

Fine-tune of openai/whisper-small for Mongolian (Cyrillic Khalkha) speech-to-text. This is the v25 run — the winning configuration of a controlled A/B against a plain fine-tuning baseline, selected on a text-disjoint evaluation set (transcripts that never appear in training) rather than the optimistic in-domain split.

• Base model: openai/whisper-small (244M params)
• Language: Mongolian, Cyrillic script (mn)
• Dataset: Blgn94/mongolian-stt-dataset-v24 (~148 h of audio)
• Honest clean WER: 21.33 (raw) / 19.67 (normalized)

Results

All numbers are Word/Character Error Rate (lower is better). Normalized = lowercased, punctuation stripped, NFC-normalized Cyrillic (ё/й) — this is what published benchmarks use, so compare against the _norm columns.

Three evaluation sets are reported:

v25 final metrics

Per-source normalized WER on the clean set: mbspeech 6.03, fleurs 21.67, youtube 57.65 (spontaneous speech is the weak spot — see Limitations).

A/B vs baseline (the reason this recipe was chosen)

Both runs share the identical text-disjoint clean eval set, so the comparison is fair.

The baseline is a plain fine-tune on the full corpus with no de-duplication or augmentation. v25 improves the honest clean WER by 3.2 points and generalizes markedly better on spontaneous YouTube speech and on real microphone recordings. The in-domain score rises — this is expected and desirable: capping duplicate transcripts removes the memorization crutch (the corpus repeats each Common Voice sentence ~14.5×), so the optimistic in-domain number falls while genuine transcription of unseen speech improves.

Training data

Blgn94/mongolian-stt-dataset-v24 — ~148 h of 16 kHz Mongolian Cyrillic speech across four sources (measured by decoding all clips):

Known weakness: the corpus is ~85% Common Voice read speech and very light on spontaneous speech (~1 h). Read-speech sources score in the single-to-low-twenties WER; spontaneous YouTube sits near 58%. Expect the model to be strongest on clear, read-style Mongolian and weaker on conversational/noisy audio.

v25 corpus shaping (what makes this run different)

• Per-sentence cap = 4: at most 4 recordings per distinct transcript, dropping 66.5% of rows (95,946 → 32,134) and cutting Common Voice text reuse from 14.5× to 4×. Breaks the transcript-memorization shortcut.
• Source-balanced sampling (α = 0.5): re-weights so Common Voice does not drown out FLEURS, MBSpeech, and YouTube.
• Audio augmentation (train only; eval always clean):
  • SpecAugment (time masking, p = 0.05)
  • Waveform augmentation (p = 0.5, up to 2 ops): speed 0.9–1.1×, gain, additive noise at 10–30 dB SNR, light reverb.

Training procedure

Model selection: best checkpoint by eval_clean_wer_norm (load_best_model_at_end). Labels drop the leading <|startoftranscript|> token (the model re-prepends decoder_start_token_id).

Reproduce

MAX_PER_SENTENCE=4 SOURCE_ALPHA=0.5 AUG_PROB=0.5 EPOCHS=4 OUT=out_v25 \
  python scripts/train.py

Full config and final metrics are in run_manifest.json in this repo.

Usage

import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="Blgn94/whisper-small-mn-v25",
    device=0 if torch.cuda.is_available() else -1,
)
result = pipe(
    "audio.mp3",
    generate_kwargs={"language": "mn", "task": "transcribe"},
)
print(result["text"])

For long audio, add chunk_length_s=30. For raw model + processor access:

from transformers import WhisperForConditionalGeneration, WhisperProcessor

processor = WhisperProcessor.from_pretrained("Blgn94/whisper-small-mn-v25")
model = WhisperForConditionalGeneration.from_pretrained("Blgn94/whisper-small-mn-v25")

Limitations

• Spontaneous / conversational speech: weak (~58% WER on YouTube), because the corpus has only ~1 h of it. Real-world microphone input is also harder (mic set ~72% WER, but only 10 clips — treat as directional, not a benchmark).
• Not evaluated on the official Common Voice test split. The clean set is a custom text-disjoint split of v24; it is not speaker-disjoint from training, so real-world WER on entirely new speakers may be somewhat higher. For a speaker-disjoint reference point, bayartsogt/whisper-small-mn-8 reports ~26.5% WER on the official Common Voice mn test.
• Cyrillic Khalkha only. No traditional Mongolian script; not tuned for other dialects.

License

Apache-2.0, inheriting from the openai/whisper-small base model. Note the underlying training data combines sources under their own licenses (Common Voice, FLEURS, MBSpeech, YouTube manual subtitles); review those before commercial use.