whisper-large-v3-turbo-lb-ct2

Whisper large-v3-turbo fine-tuned with LoRA on Luxembourgish (Lëtzebuergesch), trained only on FLEURS lb_lu (CC-BY-4.0, 8.5 h of read speech). Recipe, configuration and evaluation scripts: https://github.com/Sitraka17/lb-whisper.

What this model offers is a reproducible recipe on a small amount of open data, a robustness test on low-bitrate Opus audio, and a model that the companion meeting-notes app can run locally. On the 300-utterance comparison subset below it does not beat the uni.lu or ZLS models: both are more accurate (WER 26.1 and 28.6 against 29.5), after much more training audio.

Results

Full FLEURS test split

On all 934 utterances of the FLEURS lb_lu test split (Transformers, greedy decoding): WER 30.15 %, CER 8.82 %. Produced by finetune/evaluate.py.

Comparison with other models

All models are scored on the same audio, the same references and the same normaliser (finetune/lb_text.py: HTML entities unescaped, case and punctuation dropped, apostrophes and hyphens inside words kept). The subset is the first 300 utterances of the test split, which hold 113 distinct sentences (FLEURS records each sentence with several speakers).

• "Opus 16 kbit/s" is the WhatsApp condition: each clip is re-encoded with ffmpeg as 16 kbit/s mono Opus, then decoded back to 16 kHz PCM. Same text, lower audio quality.
• WER counts substitutions, deletions and insertions against the number of reference words, so a model that outputs more words than the reference (repetitions, hallucinated text, another language) can exceed 100 %.
• Speed is wall-clock seconds per minute of audio on the clean condition, on one GPU: batches of 16 with Transformers, one clip at a time with faster-whisper. It depends on the backend and is only a rough guide.

Caveats: according to their model cards, the uni.lu model was trained on about 150 h and the ZLS model on about 75 h of Luxembourgish; whether that data overlaps with the FLEURS sentences is unknown. The ZLS model is published only in CTranslate2 format and was decoded with faster-whisper (float16, greedy, no VAD), the others with Transformers (greedy). Treat the comparison as indicative.

Every number above is regenerated by finetune/benchmark.py, finetune/evaluate.py and finetune/fill_card.py.

Training data

The FLEURS dev split (408 utterances) was used only to select the checkpoint. The test split was not used for training or checkpoint selection. No RTL, uni.lu or ZLS audio, no scraped audio.

FLEURS attribution

This model is trained on FLEURS: A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, A. Bapna, FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, 2022, arXiv:2205.12446. Dataset: https://huggingface.co/datasets/google/fleurs (config lb_lu), licensed under CC-BY-4.0. Changes: HTML entities in the transcriptions unescaped; clips longer than 30 s removed from training.

Training

• LoRA r=32, alpha=64, dropout 0.05 on the q/k/v/out projections and both MLP layers of every block
• 3 epochs (468 steps), lr 1e-4, effective batch 16, fp16, gradient checkpointing
• one T4, 2.1 h of training
• checkpoint kept: checkpoint-468 (lowest dev WER)
• full configuration: finetune/configs/turbo-lora.yaml

Dev WER during training (400 dev utterances, same normaliser):

Usage

This repository holds the CTranslate2 int8 export of SitrakaForler/whisper-large-v3-turbo-lb, for faster-whisper. It does not load with Transformers.

from faster_whisper import WhisperModel

model = WhisperModel("SitrakaForler/whisper-large-v3-turbo-lb-ct2", device="cpu", compute_type="int8")
segments, info = model.transcribe("meeting.wav", language="lb", vad_filter=True)
for s in segments:
    print(f"[{s.start:.1f}-{s.end:.1f}] {s.text}")

The WER of this int8 export was not measured separately: the figures on this card were measured with the Transformers model SitrakaForler/whisper-large-v3-turbo-lb. VAD was not used in the benchmark.

Limitations

• Trained on 8.5 h of read speech: Wikipedia sentences read aloud one at a time.
• Not evaluated on spontaneous speech, meetings, overlapping speakers, code-switching (Luxembourgish with French, German or English in the same sentence) or speech by minors.
• Dialectal variation and the Eifeler Regel (n-deletion) are learnt only as far as FLEURS shows them.
• Numbers, dates and proper nouns follow the written conventions of the FLEURS sentences.
• Evaluated on one test set. Run finetune/benchmark.py on your own audio before relying on it.

Licence

MIT, like the base model openai/whisper-large-v3-turbo. The training data (FLEURS) is CC-BY-4.0; see the attribution above.