This repository provides a Korean dysarthria-optimized LoRA adapter (PL-HFT/whisper-large-v3-turbo-ko-dysarthria-lora) fine-tuned from openai/whisper-large-v3-turbo on a combined Korean dysarthria + typical speech corpus (combine profile from the PL-HFT project).
Single-domain fine-tuning on dysarthric speech substantially improves recognition on impaired speech, but causes severe degradation on typical speakers (domain overfitting / catastrophic forgetting). Joint 1-epoch LoRA fine-tuning on the combined corpus resolves this trade-off—improving recognition error rates across both dysarthric speakers and typical speakers simultaneously, while also improving generalization on an unseen external Korean lecture test set.
openai/whisper-large-v3-turbo (Encoder-Decoder Transformer)r=64, lora_alpha=128, lora_dropout=0.05, bias="none")q_proj, k_proj, v_proj, out_proj, fc1, fc2 (232 matched modules)ko) / Automatic Speech Recognition (transcribe)<= 29.5simport torch
import torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
base_model_id = "openai/whisper-large-v3-turbo"
adapter_id = "PL-HFT/whisper-large-v3-turbo-ko-dysarthria-lora" # or local folder path
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
dtype = torch.float16 if device in ("cuda", "mps") else torch.float32
processor = WhisperProcessor.from_pretrained(base_model_id, language="Korean", task="transcribe")
base_model = WhisperForConditionalGeneration.from_pretrained(base_model_id, torch_dtype=dtype)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval().to(device)
if hasattr(model, "generation_config") and model.generation_config is not None:
model.generation_config.forced_decoder_ids = None
# Load 16 kHz mono audio (<= 29.5s)
waveform, sr = torchaudio.load("sample.wav")
if sr != 16000:
waveform = torchaudio.functional.resample(waveform, sr, 16000)
if waveform.shape[0] > 1:
waveform = waveform.mean(dim=0, keepdim=True)
inputs = processor.feature_extractor(
waveform.squeeze(0).numpy(),
sampling_rate=16000,
return_tensors="pt",
)
input_features = inputs.input_features.to(device=device, dtype=dtype)
with torch.inference_mode():
predicted_ids = model.generate(
input_features,
language="Korean",
task="transcribe",
temperature=0.0,
condition_on_prev_tokens=False,
)
text = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(text)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./whisper-combine-merged")
processor.save_pretrained("./whisper-combine-merged")
combine Dataset)The combine corpus merges two quality-verified Korean speech datasets preprocessed by Speech-dataset-pipeline (raw audio and transcripts are not redistributed in this repository):
dysarthria, AI-Hub 013):40.66% by segment count, 44.20% by duration).kresnik/wav2vec2-large-xlsr-korean), retaining strictly aligned Green segments (<= 29.5s, natural intra-utterance pauses preserved).57.9% (164.44h), Speech & Hearing 22.3% (63.36h), Neurological 19.7% (56.05h).typical, AI-Hub 109):59.34% by segment count, 55.80% by duration).| Split | Total Segments | Total Hours | Dysarthria (Segs / Hours / Speakers) | Typical (Segs / Hours / Speakers) |
|---|---|---|---|---|
| Train | 321,135 | 514.37 h | 129,885 / 227.72 h / 617 | 191,250 / 286.65 h / 473 |
| Val | 38,459 | 64.77 h | 16,531 / 28.93 h / 74 | 21,928 / 35.83 h / 55 |
| Test | 38,599 | 63.04 h | 15,495 / 27.21 h / 75 | 23,104 / 35.83 h / 55 |
train, val, and test.cross_split_overlap_count: 55,497). Evaluation separately reports Overlapped vs. Unseen prompt subsets.NVIDIA A100-SXM4-80GB (79.3 GB VRAM)10h 00m 43s (1.0 epoch, 5,020 steps)per_device_train_batch_size=16, gradient_accumulation_steps=4)adamw_torch_fused, learning_rate=1e-4, warmup_steps=350, weight_decay=0.01, bf16=true, tf32=true, seed=42eval_steps=400, save_steps=400, load_best_model_at_end=true (metric_for_best_model=loss), Final EvalLoss=0.0596transcript_normalized (punctuation/noise tags normalized consistently across all compared models)metadata/training_config.yaml and metadata/export_manifest.json (adapter_model.safetensors SHA-256: a6c5e7b09cf64168e256f8f493bd0c3a51111a9af88acdc4bb60e4b6525a43c9).All Korean evaluations use the identical single-source normalizer (shared_controller.normalize_ko) and Levenshtein CER/WER scorer with 1,000-resample Bootstrap 95% confidence intervals.
N = 38,599)| Condition | Overall CER (95% CI) | Overall WER | Overlapped Prompts (N=34,105) CER |
Unseen Prompts (N=4,494) CER |
|---|---|---|---|---|
Base (openai/whisper-large-v3-turbo) |
2.90% [0.0273 ~ 0.0312] |
10.44% | 2.61% | 4.40% |
| Combine LoRA (This Model) | 1.17% [0.0113 ~ 0.0120] |
4.93% | 0.78% | 3.16% |
| Evaluation Set | N | Base CER / WER | Combine LoRA CER / WER | Notes |
|---|---|---|---|---|
| Dysarthria Test (All) | 15,495 | 4.45% / 13.86% | 0.59% / 1.83% | -86.7% relative CER reduction |
↳ Laryngeal (후두장애) |
11,931 | 3.56% / — | 0.34% / — | |
↳ Speech & Hearing (언어청각장애) |
1,269 | 8.82% / — | 2.13% / — | |
↳ Neurological (뇌신경장애) |
2,295 | 6.10% / — | 0.93% / — | |
| Typical Test | 23,104 | 1.89% / 8.31% | 1.53% / 6.86% | -19.0% relative CER reduction |
External Korean Lecture (external_aihub71627) |
10,000 | 14.82% / 32.37% | 11.69% / 28.80% | Out-of-domain Korean lecture speech |
3.16% CER) are higher than on overlapped prompts (0.78% CER).<= 29.5s. Non-Korean speech performance is not preserved as a primary target (on English dysarthric TORGO under a separate inference protocol, CER was 45.40% / WER 65.32%, reported as descriptive reference only).openai/whisper-large-v3-turbo (MIT License) and trained on AI-Hub 013 & AI-Hub 109 datasets. Users must comply with both the MIT license and any applicable AI-Hub data usage terms.