Qwen3-ASR-0.6B Korean Dysarthria-Optimized Merged Model (Combined Training)
Model Overview
This repository provides the standalone merged checkpoint (PL-HFT/qwen3-asr-0.6b-ko-dysarthria) created by merging PL-HFT/qwen3-asr-0.6b-ko-dysarthria-lora into Qwen/Qwen3-ASR-0.6B-hf in float32 and saving in bfloat16 (model.safetensors). Removing PEFT runtime wrapper hooks significantly speeds up startup and streaming inference.
Motivation
When Qwen3-ASR-0.6B is fine-tuned solely on dysarthric speech, its error rate on typical Korean speech degrades sharply (4.63% -> 15.01% CER in full FT, 4.63% -> 8.02% in dysarthria-only LoRA). Joint 1-epoch LoRA fine-tuning on the combined corpus eliminates this cross-domain trade-off—reducing CER on both dysarthric (4.26% -> 0.79%) and typical (4.63% -> 1.97%) speech simultaneously.
- Base Model:
Qwen/Qwen3-ASR-0.6B-hf(Qwen3-0.6B LLM + AuT audio encoder) - Merged From:
PL-HFT/qwen3-asr-0.6b-ko-dysarthria-lora(r=64,lora_alpha=128, 250 modules) - Requirements:
transformers >= 5.13.0(Qwen3ASRForConditionalGeneration), nopeftrequired - Storage Format:
model.safetensors(bfloat16, 611 tensors,1,564,928,088bytes) - Training Format: Chat-template SFT with assistant output
language Korean<asr_text>{transcript} - Audio Contract: 16 kHz mono, segment duration
<= 29.5s
Quickstart (Inference)
Important:
Qwen3-ASR-0.6B-hfrequirestransformers >= 5.13.0(withQwen3ASRForConditionalGeneration), and outputs prefixed text likelanguage Korean<asr_text>...which should be stripped after<asr_text>.
import re
import torch
import torchaudio
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration
def extract_transcript(raw_output: str) -> str:
if "<asr_text>" in raw_output:
return raw_output.split("<asr_text>", 1)[1].strip()
return re.sub(r"^language\s+(?:<[^>]+>|[A-Za-z0-9_-]+)", "", raw_output).strip()
model_id = "PL-HFT/qwen3-asr-0.6b-ko-dysarthria" # or local folder path
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
dtype = torch.bfloat16 if device in ("cuda", "mps") else torch.float32
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3ASRForConditionalGeneration.from_pretrained(model_id, dtype=dtype)
model.eval().to(device)
# Load 16 kHz mono audio (<= 29.5s)
waveform, sr = torchaudio.load("sample.wav")
if sr != 16000:
waveform = torchaudio.functional.resample(waveform, sr, 16000)
if waveform.shape[0] > 1:
waveform = waveform.mean(dim=0, keepdim=True)
audio_np = waveform.squeeze(0).numpy()
inputs = processor.apply_transcription_request(audio=audio_np, language="Korean")
prepared = {
k: (v.to(device=device, dtype=dtype) if isinstance(v, torch.Tensor) and v.is_floating_point()
else (v.to(device=device) if isinstance(v, torch.Tensor) else v))
for k, v in inputs.items()
}
with torch.inference_mode():
output_ids = model.generate(**prepared, max_new_tokens=128, do_sample=False)
prompt_len = prepared["input_ids"].shape[1]
decoded = processor.batch_decode(output_ids[:, prompt_len:], skip_special_tokens=True)[0]
print(extract_transcript(decoded))
Training Data (combine Dataset)
The combine corpus merges two quality-verified Korean speech datasets preprocessed by Speech-dataset-pipeline (raw audio and transcripts are not redistributed in this repository):
- Dysarthria Speech (
dysarthria, AI-Hub 013):- 161,911 segments / 283.86 hours (
40.66%by segment count,44.20%by duration). - Processed via Silero VAD and Korean CTC forced alignment (
kresnik/wav2vec2-large-xlsr-korean), retaining strictly aligned Green segments (<= 29.5s, natural intra-utterance pauses preserved). - Disorder breakdown (by duration): Laryngeal
57.9%(164.44h), Speech & Hearing22.3%(63.36h), Neurological19.7%(56.05h).
- 161,911 segments / 283.86 hours (
- Typical Speech (
typical, AI-Hub 109):- 236,282 segments / 358.32 hours (
59.34%by segment count,55.80%by duration).
- 236,282 segments / 358.32 hours (
Speaker-Disjoint Split Summary (Total: 398,193 segments / 642.18 hours)
| Split | Total Segments | Total Hours | Dysarthria (Segs / Hours / Speakers) | Typical (Segs / Hours / Speakers) |
|---|---|---|---|---|
| Train | 321,135 | 514.37 h | 129,885 / 227.72 h / 617 | 191,250 / 286.65 h / 473 |
| Val | 38,459 | 64.77 h | 16,531 / 28.93 h / 74 | 21,928 / 35.83 h / 55 |
| Test | 38,599 | 63.04 h | 15,495 / 27.21 h / 75 | 23,104 / 35.83 h / 55 |
- Speaker Disjointness: Zero speaker overlap across
train,val, andtest. - Prompt Overlap Audit: Read-script prompts overlap across splits (
cross_split_overlap_count: 55,497). Evaluation separately reports Overlapped vs. Unseen prompt subsets.
Training Procedure & Merge Provenance
- Hardware: 1×
NVIDIA A100-SXM4-80GB(79.3 GB VRAM) - Training Duration:
4h 36m 05s(1.0 epoch, 5,020 steps) - Effective Batch Size: 64 (
per_device_train_batch_size=16,gradient_accumulation_steps=4) - Optimizer & Schedule:
adamw_torch_fused,learning_rate=1e-4,warmup_steps=350,weight_decay=0.01,bf16=true,tf32=true,seed=42 - Model Selection:
eval_steps=400,save_steps=400,load_best_model_at_end=true(metric_for_best_model=loss), FinalEvalLoss=0.0392 - Target Text Normalization:
transcript_normalizedformatted aslanguage Korean<asr_text>{text}with assistant-only loss masking - Merge Method:
PeftModel.from_pretrained(...).merge_and_unload(safe_merge=True)infloat32, saved asbfloat16(model.safetensorsSHA-256:97b4770c35388ed676c798a1770f418c432d4e68626694028439edaf29b43ddf, max diff0.0vs.E2E-Application-CLI/caches/asr_merged) - Provenance: See
metadata/training_config.yamlandmetadata/export_manifest.json.
Evaluation Results
All Korean evaluations use the identical single-source normalizer (shared_controller.normalize_ko) and Levenshtein CER/WER scorer with 1,000-resample Bootstrap 95% confidence intervals. Benchmark numbers below were measured on the combine LoRA weights prior to merge; generation equivalence between LoRA and merged checkpoints was verified on reload.
1. Combined Internal Test Set (N = 38,599)
| Condition | Overall CER (95% CI) | Overall WER | Overlapped Prompts (N=34,105) CER |
Unseen Prompts (N=4,494) CER |
|---|---|---|---|---|
Base (Qwen/Qwen3-ASR-0.6B-hf) |
4.48% [0.0442 ~ 0.0455] |
16.00% | 4.27% | 5.61% |
| Combine LoRA / Merged (This Model) | 1.51% [0.0146 ~ 0.0156] |
5.70% | 1.02% | 4.08% |
2. Domain, Disorder Category & External Generalization Breakdown
| Evaluation Set | N | Base CER / WER | Combine CER / WER | Notes |
|---|---|---|---|---|
| Dysarthria Test (All) | 15,495 | 4.26% / 14.36% | 0.79% / 1.98% | -81.4% relative CER reduction |
↳ Laryngeal (후두장애) |
11,931 | 3.45% / — | 0.37% / — | |
↳ Speech & Hearing (언어청각장애) |
1,269 | 10.38% / — | 3.99% / — | |
↳ Neurological (뇌신경장애) |
2,295 | 4.84% / — | 1.09% / — | |
| Typical Test | 23,104 | 4.63% / 17.02% | 1.97% / 8.02% | -57.4% relative CER reduction |
External Korean Lecture (external_aihub71627) |
10,000 | 12.23% / 29.41% | 13.11% / 32.03% | Slight OOD drift vs. base |
Limitations, Bias & Safety Considerations
- Not a Standalone Clinical / Diagnostic Tool: This model is intended for speech recognition research and assistive transcription. Do not use it standalone for clinical diagnosis, severity scoring, or medical decisions. In medical intake applications (
E2E-Application), it is paired with a downstream medical LLM (google/medgemma-1.5-4b-it) for conservative error correction. - Out-of-Domain & Cross-Lingual Trade-offs: Compared with
Base,Combineachieves strong gains on both internal Korean dysarthria (0.79%) and typical (1.97%) tests, while showing mild regression on external Korean lecture audio (12.23% -> 13.11%CER) and cross-lingual English TORGO (38.13%CER /58.32%WER under a separate protocol, reported as descriptive reference only) due to Korean-focused adaptation. - Scripted-Speech Bias: Error rates are lower on overlapped read prompts (
1.02%CER) than on unseen prompts (4.08%CER). - Data Licensing: Weights inherit the Apache-2.0 license of
Qwen/Qwen3-ASR-0.6B-hf; training data derives from AI-Hub 013 & AI-Hub 109 and users must comply with applicable AI-Hub usage terms.