Whisper-large-v3 fine-tuned for Tatar (run 2)

openai/whisper-large-v3, fully fine-tuned on Common Voice Scripted Speech 27.0 Tatar (CC0). It writes Tatar in Cyrillic, with capital letters and punctuation, so its output can be used as subtitles with little editing.

This is the recommended model. It was trained on 165 speakers; run 1 was trained on 9.

• Write-up, with a comparison against existing Tatar ASR models: ifs.name/post/tatar-asr
• Demo: ifs/tatar-asr

Results

Error rates in percent on run 2's test set: 1,122 clips from 80 speakers, none of whom appear in this model's training data. CER and WER are computed after lowercasing and removing punctuation; raw CER compares the output exactly as produced.

It has the lowest raw CER of all models tested. Two wav2vec2 models have a lower normalized CER, but they write lowercase text without punctuation. The full table is in the write-up.

AigizK's language-model decoder never writes the letter й; it writes ⁇ instead, in 347 of the 1,122 clips. The table shows its output as it is; if every ⁇ is read as й, its CER drops to 2.2% and its WER to 10.3%.

This model has no score on the official Common Voice test split, because its training data includes most of that split's speakers (see below).

Usage

import torch
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="ifs/whisper-large-v3-tt-cv27-run2",
    dtype=torch.float16,   # torch.float32 on a CPU
    device="cuda",         # or "cpu"
)
result = asr(
    "audio.mp3",
    return_timestamps=True,  # needed for audio longer than 30 seconds
    generate_kwargs={
        "language": "tatar",
        "task": "transcribe",
        "num_beams": 1,
        "no_repeat_ngram_size": 3,
        "repetition_penalty": 1.2,
    },
)
print(result["text"])

The decoding settings are the ones used in evaluation; they follow BuzzASR. The weights are stored in float32 (6.2 GB); float16 halves the memory and works well for inference.

Training data

All clips come from the validated bucket of Common Voice Scripted Speech 27.0 Tatar. The official splits put 258 of 270 speakers in the test split and leave 2 for training, so this run re-split the data:

One speaker accounts for 52% of the training clips. The exact splits are in the splits/ folder of this repository. Labels keep their original casing and punctuation.

Training procedure

Full fine-tuning following the BuzzASR recipe, with finetune.py --resplit:

• learning rate 6.48e-6 for the decoder, 0.3 times that for the encoder
• cosine schedule, 150 warm-up steps, weight decay 4e-5
• effective batch size 32 (4 clips × 8 gradient accumulation steps)
• bf16 mixed precision, with the weights loaded in float32
• evaluation on dev CER, early stopping with patience 3, best checkpoint kept
• seed 42

Hardware: one NVIDIA A100 80GB on RunPod. Software: PyTorch 2.8.0, Transformers 5.18.0 (full environment in the write-up).

Limitations

• Evaluated on read speech only. Accuracy on spontaneous speech, such as YouTube videos or podcasts, is untested and likely lower.
• One speaker contributes about half of the training data.
• Code-switching into Russian and timestamp accuracy are untested.

Licence

MIT, like the base model. The training data is CC0.

Author

Ilnar Salimzianov