Whisper Base for Karakalpak (Qaraqalpaqsha), distilled
📄 Paper: Karakalpak Speech Recognition with Whisper: An Open 107-Hour Corpus, a Held-Out Benchmark, and Knowledge Distillation into Smaller Models (Zenodo, DOI 10.5281/zenodo.23291730)
openai/whisper-base (73M parameters) distilled from atikuwu/whisper-medium-karakalpak on the training split of the Karakalpak Speech Corpus. The smallest and fastest of our models, about 7× faster than the medium model on CPU.
| Params | Test WER (95% CI), greedy | Test CER | CPU real-time factor* | |
|---|---|---|---|---|
| whisper-medium-karakalpak (teacher) | 764M | 10.66 (10.03–11.36) | 3.83 | 1.28 |
| whisper-small-karakalpak (distilled) | 242M | 11.52 (10.84–12.26) | 4.03 | 0.47 |
| this model (distilled) | 73M | 15.54 (14.57–16.79) | 5.71 | 0.19 |
| whisper-base fine-tuned directly, same recipe | 73M | 18.06 (16.43–20.30) | 6.95 | 0.19 |
*processing time ÷ audio length, 4 CPU cores, batch size 1, greedy. Below 1 = faster than real time.
Distillation beats fine-tuning the same model directly by 2.52 WER (95% CI 1.36–4.35, p < 0.001). Validation WER: 14.54 (CER 5.24).
Training
- Teacher: atikuwu/whisper-medium-karakalpak
- Loss: cross-entropy on the teacher's greedy transcripts of the training audio + 0.8 · T² · KL to the teacher's token distribution (T = 2), as in Distil-Whisper. Utterances whose teacher transcript has WER > 20% against the human transcript were dropped (19,178 of 19,255 kept).
- Recipe: 6 epochs, effective batch 32, learning rate 5e-5 with 500 warm-up steps, 8-bit AdamW, fp16, SpecAugment, best checkpoint by validation WER
- Hardware: 2× NVIDIA T4 (Kaggle), 3.2 h
- Language token:
uz
Usage
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="atikuwu/whisper-base-karakalpak")
result = pipe("sample_audio.ogg", generate_kwargs={"task": "transcribe", "language": "uz"})
print(result["text"])
Beam search and the 4-gram language model from whisper-medium-karakalpak also work with this model, but we have not tuned or evaluated them for it.
Team
Atabek Kadirbergenov, Sanjar Tleumuratov, Dauitbay Nasiredinov, Shohjahon Isroilov, Rasul Zinatdinov (Muhammad al-Khwarizmi Specialized School, Nukus), with 200+ volunteer speakers. See the medium model card for details and citation.