This repository provides a Korean dysarthria-targeted LoRA adapter (PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria-lora) fine-tuned from nvidia/nemotron-3.5-asr-streaming-0.6b on a combined Korean dysarthria + typical speech corpus (combine profile from the PL-HFT project).
Important Loading & Convergence Notice:
- Unlike standard Hugging Face
PeftModeladapters, this adapter was injected into a NeMoEncDecRNNTBPEModelviapeft.inject_adapter_in_model(...). Use the custom NeMo + PEFT loading snippet below rather thanPeftModel.from_pretrained(...).- During the 1-epoch LoRA run, the NeMo scheduler held the learning rate at
min_lr = 1.0e-06after warmup, resulting in conservative weight updates and modest improvement over the zero-shot baseline (7.11% -> 6.82%CER on Combined Test). See Limitations for full details.
nvidia/nemotron-3.5-asr-streaming-0.6b (Cache-Aware FastConformer 24-layer + RNNT decoder)att_context_size=[56, 13], target_lang="ko-KR"inject_adapter_in_model (r=64, lora_alpha=128, lora_dropout=0.05, bias="none")linear_q, linear_k, linear_v, linear_out, linear_fc1, linear_fc2 (96 matched modules)adapter_model.safetensors (recommended for safe loading) and the original adapter_model.pt (for direct compatibility with Nemotron35-asr-ft/src/evaluate.py).<= 29.5spip install torch torchaudio peft safetensors huggingface_hub
pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/NeMo.git@main"
from pathlib import Path
import re
import nemo.collections.asr as nemo_asr
from huggingface_hub import hf_hub_download
from peft import LoraConfig, inject_adapter_in_model, set_peft_model_state_dict
from safetensors.torch import load_file
base_model_id = "nvidia/nemotron-3.5-asr-streaming-0.6b"
adapter_repo_id = "PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria-lora"
# 1. Load NeMo base model
model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(base_model_id)
# 2. Download adapter config & safetensors weights
cfg_path = hf_hub_download(repo_id=adapter_repo_id, filename="adapter_config.json")
weights_path = hf_hub_download(repo_id=adapter_repo_id, filename="adapter_model.safetensors")
lora_cfg = LoraConfig.from_pretrained(str(Path(cfg_path).parent))
inject_adapter_in_model(lora_cfg, model)
adapter_state = load_file(weights_path)
set_peft_model_state_dict(model, adapter_state)
# 3. Configure streaming context and Korean prompt
if hasattr(model, "encoder") and hasattr(model.encoder, "set_default_att_context_size"):
model.encoder.set_default_att_context_size([56, 13])
if hasattr(model, "set_inference_prompt"):
model.set_inference_prompt("ko-KR")
model.eval()
# 4. Transcribe 16 kHz mono audio (strip Nemotron language/control tags such as <ko-KR>)
raw_preds = model.transcribe(["sample.wav"], batch_size=1)
first = raw_preds[0]
pred_text = first.text if hasattr(first, "text") else str(first)
clean_text = re.sub(r"<[^>]+>", "", pred_text).strip()
print(clean_text)
If loading locally in
Nemotron35-asr-ft/src/evaluate.py, pass the repository folder path containingadapter_config.jsonandadapter_model.ptvia--checkpoint.
combine Dataset)The combine corpus merges two quality-verified Korean speech datasets preprocessed by Speech-dataset-pipeline (raw audio and transcripts are not redistributed in this repository):
dysarthria, AI-Hub 013):40.66% by segment count, 44.20% by duration).kresnik/wav2vec2-large-xlsr-korean), retaining strictly aligned Green segments (<= 29.5s, natural intra-utterance pauses preserved).57.9% (164.44h), Speech & Hearing 22.3% (63.36h), Neurological 19.7% (56.05h).typical, AI-Hub 109):59.34% by segment count, 55.80% by duration).| Split | Total Segments | Total Hours | Dysarthria (Segs / Hours / Speakers) | Typical (Segs / Hours / Speakers) |
|---|---|---|---|---|
| Train | 321,135 | 514.37 h | 129,885 / 227.72 h / 617 | 191,250 / 286.65 h / 473 |
| Val | 38,459 | 64.77 h | 16,531 / 28.93 h / 74 | 21,928 / 35.83 h / 55 |
| Test | 38,599 | 63.04 h | 15,495 / 27.21 h / 75 | 23,104 / 35.83 h / 55 |
train, val, and test.cross_split_overlap_count: 55,497). Evaluation separately reports Overlapped vs. Unseen prompt subsets.NVIDIA A100-SXM4-80GB (79.3 GB VRAM)2h 08m 54s (1.0 epoch, 5,020 steps)per_device_train_batch_size=16, gradient_accumulation_steps=4)bf16-mixed, tf32=true, gradient_clip_val=1.0, configured learning_rate=1e-4, min_lr=1.0e-06, warmup_steps=350, weight_decay=0.01, seed=42monitor=val_wer, mode=min, early_stopping_patience=4, eval_steps=400, save_steps=400metadata/training_config.yaml and metadata/export_manifest.json (adapter_model.pt SHA-256: 72e49e3b41f9db406c4a6d513f4dd076b57a9654d59d8453db59ac503cb2494f).All Korean evaluations use the identical single-source normalizer (shared_controller.normalize_ko) and Levenshtein CER/WER scorer with 1,000-resample Bootstrap 95% confidence intervals (att_context_size=[56, 13], target_lang="ko-KR").
N = 38,599)| Condition | Overall CER (95% CI) | Overall WER | Overlapped Prompts (N=34,105) CER |
Unseen Prompts (N=4,494) CER |
|---|---|---|---|---|
Base (nvidia/nemotron-3.5-asr-streaming-0.6b) |
7.11% [0.0703 ~ 0.0720] |
24.01% | 7.02% | 7.60% |
| Combine LoRA (This Model) | 6.82% [0.0674 ~ 0.0691] |
23.75% | 6.71% | 7.40% |
| Evaluation Set | N | Base CER / WER | Combine LoRA CER / WER | Notes |
|---|---|---|---|---|
| Dysarthria Test (All) | 15,495 | 7.42% / 24.75% | 7.22% (7.24%*) / 24.25% | Slight improvement over base |
↳ Laryngeal (후두장애) |
11,931 | 6.05% / — | 5.89% / — | |
↳ Speech & Hearing (언어청각장애) |
1,269 | 15.77% / — | 15.19% / — | |
↳ Neurological (뇌신경장애) |
2,295 | 9.10% / — | 9.03% / — | |
| Typical Test | 23,104 | 6.92% / 23.57% | 6.57% (6.55%*) / 23.42% | Slight improvement over base |
External Korean Lecture (external_aihub71627) |
10,000 | 14.02% / 35.00% | 14.08% / 34.97% | Comparable to base |
*Note: 7.22% / 6.57% are from the 03_eval_finetuned.ipynb evaluation log; 7.24% / 6.55% are from the harmonized figure table (asr_visualization_scores.csv).
min_lr = 1.0e-06 after warmup, resulting in minimal weight divergence from the base checkpoint (7.11% -> 6.82% CER overall). For stronger dysarthric adaptation with Nemotron-3.5-ASR, full fine-tuning (dysarthria Full FT reached 3.92% CER) or tuning the NeMo scheduler configuration is recommended.inject_adapter_in_model was used without wrapping in PeftModel, standard PeftModel.from_pretrained() does not load this checkpoint directly; use the NeMo + PEFT script above.N=2,847, separate inference protocol, descriptive reference only): 43.47% CER / 54.91% WER (vs. legacy Base 43.23% / 55.07%).nvidia/nemotron-3.5-asr-streaming-0.6b under OpenMDW-1.1; retain OpenMDW-1.1 notices on redistribution and comply with AI-Hub 013 & AI-Hub 109 data usage terms.adapter_model.pt 50 MBadapter_model.safetensors 50 MB