whisper-medium-sk

Slovak fine-tune of openai/whisper-medium by the KInIT team. Full parameter fine-tuning on a curated Slovak speech corpus, with noise augmentation for robustness to real-world recording conditions. Listed in SLAIH, a catalog of Slovak NLP resources.

Model Details

Intended Use

This model is intended for Slovak automatic speech recognition across a range of domains and recording conditions.

Out-of-scope: Non-Slovak audio, real-time streaming without appropriate chunking, safety-critical transcription without human review.

Evaluation

Word Error Rate (WER) and Character Error Rate (CER), lower is better. Measured on two Slovak eval sets:

• CV24 - Common Voice 24.0 test split (5,239 samples, public)
• Internal - held-out KInIT set (9,317 samples, stratified by domain and speaker gender, one third clean and two thirds noise-augmented; not public)

Before vs. after fine-tuning

Slovak fine-tuning reduces WER by 71% on Common Voice and 78% on the internal eval set, at the cost of multilingual performance (see Limitations).

Choosing a model size

Comparison of all six Whisper sizes before and after Slovak fine-tuning. WER is measured on our internal benchmark dataset (see above):

The full KInIT ASR collection includes all six Whisper sizes plus the NVIDIA NeMo fine-tunes (Canary, Parakeet).

Training Data

Fine-tuned on an internal curated Slovak speech corpus compiled at KInIT. The corpus combines public datasets with internal KInIT recordings. Recordings containing personal data were anonymised prior to use. Samples were quality-filtered using a CER-based threshold validated against multiple ASR models. During training, 75% of batches were additionally augmented on the fly with synthetic background noise for robustness to real-world recording conditions (see Training Procedure below).

Training Procedure

Training was performed on the Devana HPC cluster.

Usage

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "kinit/whisper-medium-sk"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    dtype=dtype,
    use_safetensors=True,
).to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)

result = pipe("audio.wav", generate_kwargs={"language": "slovak"})
print(result["text"])

License

Apache 2.0

Limitations

• Catastrophic forgetting: Fine-tuning exclusively on Slovak data significantly degrades performance on other languages. Use the base openai/whisper-medium if multilingual transcription is required.
• Performance may degrade on strongly accented, dialectal, or domain-specific speech not represented in the training data.
• Maximum reliable single-segment length is 30 seconds without chunking.

Acknowledgements

Public datasets used in training: SloPalSpeech, Mozilla Common Voice, FLEURS, and the TEDxSK and JumpSK Lecture Speech Corpus (KEMT NLP).

(Part of the) Research results was obtained using the computational resources procured in the national project National competence centre for high performance computing (project code: 311070AKF2) funded by European Regional Development Fund, EU Structural Funds Informatization of society, Operational Program Integrated Infrastructure.