whisper-small-vi-finetuned

A Vietnamese automatic speech recognition model, fine-tuned from openai/whisper-small (244M parameters) on natmin322/18k_vietnamese_voice_augmented_of_VigBigData.

Trained entirely on Kaggle across two resumed sessions within a 12-hour/day dual-T4 GPU budget.

Results

Measured against the unmodified base model on a held-out 500-sample test subset:

A 53% relative reduction in Word Error Rate and 57% in Character Error Rate.

Beyond the metrics, two qualitative changes matter. The base model occasionally collapses into a repetition loop, emitting the same phrase until it hits the generation length limit; the fine-tuned model did not exhibit this on any inspected sample. It also adopted the corpus's orthographic conventions — see Output format below.

Usage

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="qa2code/whisper-small-vi-finetuned",
    device=0,  # omit for CPU
)

result = pipe("audio.wav", generate_kwargs={"language": "vi", "task": "transcribe"})
print(result["text"])

Audio is resampled to 16 kHz internally. Passing language="vi" explicitly is recommended — without it, Whisper's language detection can misfire on short or noisy clips and decode into the wrong language.

For clips longer than 30 seconds, enable chunked long-form decoding:

result = pipe(
    "long_audio.wav",
    chunk_length_s=30,
    stride_length_s=5,
    generate_kwargs={"language": "vi", "task": "transcribe"},
)

Loading the model and processor directly:

from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

processor = AutoProcessor.from_pretrained("qa2code/whisper-small-vi-finetuned")
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "qa2code/whisper-small-vi-finetuned", device_map="auto"
)

Output format

The model inherits the transcription conventions of its training corpus. Output is:

• lowercase throughout — no sentence-initial capitals, no capitalised proper nouns
• without terminal punctuation — no full stops, commas, or question marks
• numbers spelled as words — mười tám tháng mười, not 18 tháng 10

This is a deliberate property of the data, not a defect, but it means the raw output is not display-ready. Downstream applications needing conventional orthography should add a punctuation-and-capitalisation restoration step.

Training data

The source dataset contains 18,005 audio–transcript pairs at 22.05 kHz. Exploratory analysis showed a heavily right-skewed duration distribution (median 5.76 s, mean 6.92 s, tail to 80 s) and residual <unk> markers left over from the upstream pipeline.

A cleaning pass applied three filters:

1. Keep clips with duration in [1 s, 25 s] — shorter clips carry little linguistic content, longer ones exceed Whisper's 30-second receptive field and waste VRAM on padding.
2. Strip <unk> markers and collapse the resulting whitespace.
3. Drop transcripts shorter than two words.

Result: 18,005 → 17,169 samples (9,695 train / 4,647 validation / 2,827 test).

Training procedure

Training was split across two Kaggle sessions to fit the 12-hour daily dual-T4 quota. The first session ran to step 1,000 and pushed its checkpoint to the Hub; the second resumed from checkpoint-1000 with max_steps raised to 3,000, restoring optimizer state, scheduler position, and RNG state. The run is therefore continuous rather than two independent trainings.

Validation during training used a fixed 500-sample subset to bound evaluation cost.

Training results

Limitations

In-domain evaluation only. The 31.74 WER figure is measured on a held-out split of the same corpus the model was trained on — read speech, studio-adjacent conditions, augmented. It is not directly comparable to numbers reported on Common Voice, VIVOS, or FLEURS, and real-world performance on spontaneous speech, regional accents, telephone-bandwidth audio, or noisy environments will be meaningfully worse. Benchmark on your own data before deploying.

Undertrained by design. 3,000 steps is a modest budget for a 244M-parameter encoder–decoder, and the loss curve had not plateaued when the run was stopped by the hardware quota. This result is closer to a floor than a ceiling.

No punctuation or casing, as described in Output format.

Inherits Whisper's failure modes, including hallucination on silence and non-speech audio. Voice activity detection upstream is advisable for production use.

Directions for improvement

• Extend training past 3,000 steps — the largest single gain still available
• Add SpecAugment for regularisation
• Evaluate on out-of-domain Vietnamese benchmarks to measure generalisation
• Explore LoRA/PEFT to fit whisper-medium or whisper-large-v3 into the same VRAM envelope

Framework versions

• Transformers 5.0.0
• PyTorch 2.10.0+cu128
• Datasets 5.0.0
• Tokenizers 0.22.2

License

Apache 2.0, inherited from openai/whisper-small. Note that the training dataset carries its own licence terms — check the dataset card before commercial use.

Citation

@misc{whisper-small-vi-finetuned,
  author = {Nguyen, Quoc Anh},
  title  = {whisper-small-vi-finetuned: Vietnamese ASR fine-tuned from Whisper-small},
  year   = {2026},
  url    = {https://huggingface.co/qa2code/whisper-small-vi-finetuned}
}