Whisper Large-v3-Turbo Swiss German — commercially usable (MIT)

Swiss German speech to Standard German text. LoRA fine-tune of openai/whisper-large-v3-turbo, merged into the weights, with a deliberately clean licence chain: MIT base model, MIT training data, MIT result.

Most public Swiss German models inherit CC BY-NC from SwissDial and cannot be used commercially. This checkpoint avoids that: it is trained only on MIT-licensed FHNW/i4ds material.

Results

Measured on the FHNW/i4ds All Swiss German Dialects Test Set (MIT, 5750 clips, 12.7 h, 17 dialects), decoded with faster-whisper (CTranslate2, float16, beam 5) on an RTX 3080 Ti.

2.8 WER points better than the untouched base model at identical speed.

The test set is split speaker-disjointly into two halves: dev (2930 clips) was used for model selection, final (2820 clips) was held out. The final half is harder for every model measured, including untrained baselines — compare within a column, not across. The test set was never trained on.

WER/CER are per-clip means after normalisation (lowercase, punctuation removed, ß → ss).

Usage

faster-whisper / CTranslate2 — weights are in the ct2/ subfolder, so fetch that folder first (the repository root holds the transformers format, whose config.json differs):

from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel

path = snapshot_download("gcoli/whisper-large-v3-turbo-swiss-german-mit",
                         allow_patterns="ct2/*")
model = WhisperModel(f"{path}/ct2", device="cuda", compute_type="float16")
segments, info = model.transcribe("input.wav", language="de", beam_size=5)
print(" ".join(s.text.strip() for s in segments))

transformers — the repository root:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="gcoli/whisper-large-v3-turbo-swiss-german-mit", device=0)
print(asr("input.wav", generate_kwargs={"language": "german", "task": "transcribe"})["text"])

Use language token de and task transcribe: the model outputs Standard German, not dialect spelling. Audio should be 16 kHz mono.

Long recordings

On recordings of several minutes, turn context carry-over off. faster-whisper leaves condition_on_previous_text on by default, and on 15-minute files it cost this model family 2.5 (turbo) to 11.7 (large) WER points in our measurements, with the output growing ~14 % longer than the reference — the model starts inventing text. VAD filtering did not help either.

segments, info = model.transcribe("meeting.wav", language="de", beam_size=5,
                                  condition_on_previous_text=False, vad_filter=False)

Measured on 26 files of ~15 min each, built by concatenating the held-out half of the test set, so the audio content is identical to the short-form numbers above. Expect roughly 1 to 3 WER points worse on long recordings than on single sentences; that gap differs per model and does not follow the short-form ranking.

Compatibility

• Format: safetensors (fp16) at the repository root, CTranslate2 float16 under ct2/
• Runtimes: transformers, faster-whisper / CTranslate2
• int8_float16 quantisation costs about 0.3 WER points on this checkpoint's base family and is a reasonable trade for VRAM-constrained setups
• For MLX (Apple Silicon), convert the root safetensors with mlx_whisper

Training

The Zürich corpus is audio only. Transcripts were generated with openai/whisper-large-v3 (Apache-2.0) and filtered by confidence (avg_logprob > -0.45, no_speech_prob < 0.3, compression ratio < 2.2, repetition-loop rejection).

Two findings from the run series this checkpoint comes from:

• Learning rate dominates. The same recipe at lr 1e-3 produced no gain at all over the base model; 3e-4 gained 1.8 points immediately.
• Source mixture matters more than volume. Both corpora must be present — Bern-only and Zürich-only fine-tunes are clearly worse — but the ratio between 50 % and 89 % Zürich is not measurably different, and 80 h performs as well as 160 h or 247 h.

Other languages

Fine-tuning on one language variety can damage the others. Measured on google/fleurs (250 validation clips per language, no prompt, beam 5):

The fine-tune costs 0.3 to 1.0 WER points in the other languages — a real but mild regression, not catastrophic forgetting. If you need a multilingual system, keep the base model for everything except Swiss German.

Limitations

• The test set is read speech; the training data is parliamentary speech. Performance on spontaneous conversation, podcasts or noisy recordings is unmeasured.
• Training data covers two regions only (Bern, Zürich). Valais and Fribourg remain the weakest dialects at roughly 30–35 % WER.
• Zürich transcripts are pseudo-labels: the student cannot substantially exceed its teacher.
• Whisper can hallucinate or omit text, particularly with noise, silence, overlapping speakers, uncommon dialects or specialised vocabulary. Do not use its output as the sole basis for high-impact decisions. Obtain consent before transcribing people.

Provenance

• Base: openai/whisper-large-v3-turbo
• Pseudo-label teacher: openai/whisper-large-v3 (Apache-2.0)
• Training data: FHNW/i4ds Swiss Parliaments Corpus v2 and Gemeinderat Zürich Audio Corpus, both MIT, https://www.cs.technik.fhnw.ch/i4ds-datasets
• Evaluation: FHNW/i4ds All Swiss German Dialects Test Set (MIT), same source
• Details in training_provenance.json

License and usage conditions

MIT. The base model (openai/whisper-large-v3-turbo) is MIT, the training data (FHNW/i4ds Swiss Parliaments, Gemeinderat Zürich) is MIT, and the pseudo-label teacher (openai/whisper-large-v3) is Apache-2.0. Commercial use is permitted.

Please credit FHNW/i4ds for the underlying corpora. This repository grants no additional rights to upstream models, training data or software, and implies no endorsement by FHNW, i4ds or OpenAI.