whisper-large-v3-turbo-danish

  ░▒▓█╔════════════════════════════════════════════════════════════════════╗█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█│       ██╗    ██╗██╗  ██╗██╗███████╗██████╗ ███████╗██████╗         │█▓▒░
  ░▒▓█│       ██║    ██║██║  ██║██║██╔════╝██╔══██╗██╔════╝██╔══██╗        │█▓▒░
  ░▒▓█│       ██║ █╗ ██║███████║██║███████╗██████╔╝█████╗  ██████╔╝        │█▓▒░
  ░▒▓█│       ██║███╗██║██╔══██║██║╚════██║██╔═══╝ ██╔══╝  ██╔══██╗        │█▓▒░
  ░▒▓█│       ╚███╔███╔╝██║  ██║██║███████║██║     ███████╗██║  ██║        │█▓▒░
  ░▒▓█│        ╚══╝╚══╝ ╚═╝  ╚═╝╚═╝╚══════╝╚═╝     ╚══════╝╚═╝  ╚═╝        │█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█│             ░░▒▒▓▓  D A N I S H   S P E E C H  ▓▓▒▒░░              │█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
  ░▒▓█│  R E L E A S E   I N F O                      [ MODEL / WEIGHTS ]  │█▓▒░
  ░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█│  EDITION .......... LARGE-V3-TURBO                                 │█▓▒░
  ░▒▓█│  LANGUAGE ......... DANISH                                         │█▓▒░
  ░▒▓█│  STATUS ........... SOTA-ish                                       │█▓▒░
  ░▒▓█│  LAYERS ........... 32 ENCODER / 4 DECODER                         │█▓▒░
  ░▒▓█│  MODEL WIDTH ...... 1280                                           │█▓▒░
  ░▒▓█│  AUDIO INPUT ...... 16 kHz MONO / 128 MEL BINS                     │█▓▒░
  ░▒▓█│  TRAIN WINDOW ..... 30 SECONDS                                     │█▓▒░
  ░▒▓█│  VOCABULARY ....... 51,866 TOKENS                                  │█▓▒░
  ░▒▓█│  OPTIMIZER ........ MUON + ADAMW                                   │█▓▒░
  ░▒▓█│  TRAINING ......... 200,000 OPTIMIZER STEPS                        │█▓▒░
  ░▒▓█│  TRAINED ON ....... NVIDIA GEFORCE RTX 4090                        │█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
  ░▒▓█│             01 USAGE  /  02 TRAINING  /  03 EVALUATION             │█▓▒░
  ░▒▓█│            04 LIMITATIONS  /  05 LICENSE  /  06 CREDITS            │█▓▒░
  ░▒▓█│                                                                    │█▓▒░
  ░▒▓█╚════════════════════════════════════════════════════════════════════╝█▓▒░
     ░░▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒░░

This is a Danish continuation of openai/whisper-large-v3-turbo. The architecture and tokenizer are unchanged. The released weights are an exponential moving average taken after 200,000 optimizer steps.

01 / USAGE

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "thorh/whisper-large-v3-turbo-danish"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    dtype=dtype,
    low_cpu_mem_usage=True,
).to(device)

transcribe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)

result = transcribe(
    "audio.wav",
    generate_kwargs={"language": "da", "task": "transcribe", "num_beams": 1},
)
print(result["text"])

02 / TRAINING

Training used the Danish portions of CoRal v3, Common Voice 26.0, FLEURS, FTSpeech and NST together with filtered pseudo-labelled data. The pseudo-labels were generated with openai/whisper-large-v3 and confidence-filtered before use.

Targets retained their original casing, punctuation and numeral style. Audio augmentation, SpecAugment and synthetic no-speech examples were included. The data design was inspired by the weak-supervision and no-speech principles in the original Whisper paper.

Setting	Value
Steps	200,000
Optimizer	Muon for hidden matrices; AdamW for remaining parameters
Peak learning rate	Muon `5e-6`; AdamW `2.5e-6`
Weight decay	`0.01`
Schedule	1,000-step warmup, stable to 50,000, linear decay to 10% at 200,000
EMA	`0.9999` per step
Precision	BF16 parameters, FP8 hidden forward GEMMs, compensated FP32 optimizer updates
Maximum training segment	30 seconds
Training hardware	One NVIDIA GeForce RTX 4090
Active wall time	Approximately 56 hours

The following figures are cumulative audio sampled during training. They include repeated sampling and are not unique dataset durations.

Source	Hours seen
CoRal conversation	485.45
CoRal read-aloud	894.45
Common Voice Danish	72.63
FLEURS Danish	74.06
FTSpeech	1,469.10
NST Danish	712.01
Pseudo-labelled data	9,200.04
Total	12,907.73

Additional recipe details are recorded in training_recipe.json.

03 / EVALUATION

The checkpoint was evaluated against the public Danish ASR Leaderboard protocol using greedy Danish transcription. Short clips used standard decoding; clips over 30 seconds used timestamped sequential long-form decoding. The leaderboard normalizer was applied, including Danish numeral normalization.

Test set	Samples	WER (%)	CER (%)
CoRal conversation	8,438	20.05	11.92
CoRal read-aloud	9,122	12.93	4.80
Common Voice (`cv17_da`)	2,756	8.24	2.65
FLEURS `da_dk`	930	8.95	3.65
FTSpeech `test_balanced`	5,534	7.39	4.19
Unweighted domain mean	26,780	11.51	5.44

Exact revisions and decoding settings are recorded in training_recipe.json.

04 / LIMITATIONS

Accuracy in languages other than Danish may be reduced.
Long-form behavior was not directly trained and should be validated separately.

05 / LICENSE

The model weights are released under the MIT License. Training data are not redistributed and remain subject to their source terms. See THIRD_PARTY_NOTICES.md.

06 / CREDITS

@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}

thorhojhus/whisper-large-v3-turbo-danish