Whisper Small for Karakalpak (Qaraqalpaqsha), distilled

📄 Paper: Karakalpak Speech Recognition with Whisper: An Open 107-Hour Corpus, a Held-Out Benchmark, and Knowledge Distillation into Smaller Models (Zenodo, DOI 10.5281/zenodo.23291730)

openai/whisper-small (242M parameters) distilled from atikuwu/whisper-medium-karakalpak on the training split of the Karakalpak Speech Corpus. It is 3× smaller than the medium model and runs faster than real time on a 4-core CPU.

With beam search and the LM it is statistically indistinguishable from the medium model with greedy decoding (10.66%, paired bootstrap p = 0.20).

*processing time ÷ audio length, 4 CPU cores, batch size 1, greedy. Below 1 = faster than real time.

Distillation beats fine-tuning the same model directly on the human transcripts by 0.75 WER (95% CI 0.14–1.27, p = 0.02).

Training

• Teacher: atikuwu/whisper-medium-karakalpak
• Loss: cross-entropy on the teacher's greedy transcripts of the training audio + 0.8 · T² · KL to the teacher's token distribution (T = 2), as in Distil-Whisper. Utterances whose teacher transcript has WER > 20% against the human transcript were dropped (19,178 of 19,255 kept).
• Recipe: 6 epochs, effective batch 32, learning rate 3e-5 with 500 warm-up steps, 8-bit AdamW, fp16, SpecAugment, best checkpoint by validation WER
• Hardware: 2× NVIDIA T4 (Kaggle), 4.9 h
• Language token: uz

Evaluation: same normalisation as the medium model card (lower-case, punctuation removed, byline ÓzA removed), test split scored once.

Usage

import torch
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="atikuwu/whisper-small-karakalpak",
                device="cuda:0" if torch.cuda.is_available() else "cpu")
result = pipe("sample_audio.ogg", generate_kwargs={"task": "transcribe", "language": "uz"})
print(result["text"])

For beam search with the 4-gram language model, use lm/kaa-4gram.arpa and code/kaa_asr.py from whisper-medium-karakalpak with alpha=0.25, beta=2.0 (tuned on validation for this model).

Team

Atabek Kadirbergenov, Sanjar Tleumuratov, Dauitbay Nasiredinov, Shohjahon Isroilov, Rasul Zinatdinov (Muhammad al-Khwarizmi Specialized School, Nukus), with 200+ volunteer speakers. See the medium model card for details and citation.