This repository provides the standalone merged checkpoint (PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria) created by merging PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria-lora into nvidia/nemotron-3.5-asr-streaming-0.6b. Unlike the adapter repository, no peft injection step is required—the repository distributes both a standalone NeMo archive (.nemo) and a Hugging Face transformers checkpoint (model.safetensors for Nemotron3_5AsrForRNNT), verified bit-for-bit identical across all 96 merged self-attention projections (max_diff = 0.0).
Important Convergence Notice: During the underlying 1-epoch LoRA run, the NeMo scheduler held the learning rate at
min_lr = 1.0e-06after warmup, resulting in conservative weight updates and modest improvement over the zero-shot baseline (7.11% -> 6.82%CER on Combined Test). See Limitations for full details.
nvidia/nemotron-3.5-asr-streaming-0.6b (Cache-Aware FastConformer 24-layer + RNNT decoder)PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria-lora (r=64, lora_alpha=128, 96 self_attn projections)att_context_size=[56, 13] (num_lookahead_tokens=13), target_lang="ko-KR"transformers: model.safetensors (float32, 655 tensors, 2,552,062,944 bytes) + config.json / processor_config.json / tokenizer.json (transformers >= 5.20 with Nemotron3_5AsrForRNNT)nemotron-3.5-asr-streaming-0.6b-ko-dysarthria.nemo (2,553,098,240 bytes, standalone without PEFT layers)<= 29.5stransformers (No NeMo / PEFT Required)import re
import torch
import torchaudio
from transformers import AutoProcessor, Nemotron3_5AsrForRNNT
model_id = "PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria"
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
processor = AutoProcessor.from_pretrained(model_id)
processor.set_num_lookahead_tokens(13) # matches att_context_size=[56, 13]
model = Nemotron3_5AsrForRNNT.from_pretrained(model_id)
model.eval().to(device)
# Load 16 kHz mono audio (<= 29.5s)
waveform, sr = torchaudio.load("sample.wav")
if sr != 16000:
waveform = torchaudio.functional.resample(waveform, sr, 16000)
if waveform.shape[0] > 1:
waveform = waveform.mean(dim=0, keepdim=True)
inputs = processor(audio=waveform.squeeze(0).numpy(), sampling_rate=16000, language="ko-KR")
prepared = {k: (v.to(device) if isinstance(v, torch.Tensor) else v) for k, v in inputs.items()}
with torch.inference_mode():
generated = model.generate(**prepared)
raw_text = processor.batch_decode(generated.sequences, skip_special_tokens=True)[0]
clean_text = re.sub(r"<[^>]+>", "", raw_text).strip()
print(clean_text)
.nemo Archive, No PEFT Required)pip install torch torchaudio
pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/NeMo.git@main"
import re
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(
"PL-HFT/nemotron-3.5-asr-streaming-0.6b-ko-dysarthria"
)
if hasattr(model, "encoder") and hasattr(model.encoder, "set_default_att_context_size"):
model.encoder.set_default_att_context_size([56, 13])
if hasattr(model, "set_inference_prompt"):
model.set_inference_prompt("ko-KR")
model.eval()
raw_preds = model.transcribe(["sample.wav"], batch_size=1)
first = raw_preds[0]
pred_text = first.text if hasattr(first, "text") else str(first)
clean_text = re.sub(r"<[^>]+>", "", pred_text).strip()
print(clean_text)
combine Dataset)The combine corpus merges two quality-verified Korean speech datasets preprocessed by Speech-dataset-pipeline (raw audio and transcripts are not redistributed in this repository):
dysarthria, AI-Hub 013):40.66% by segment count, 44.20% by duration).kresnik/wav2vec2-large-xlsr-korean), retaining strictly aligned Green segments (<= 29.5s, natural intra-utterance pauses preserved).57.9% (164.44h), Speech & Hearing 22.3% (63.36h), Neurological 19.7% (56.05h).typical, AI-Hub 109):59.34% by segment count, 55.80% by duration).| Split | Total Segments | Total Hours | Dysarthria (Segs / Hours / Speakers) | Typical (Segs / Hours / Speakers) |
|---|---|---|---|---|
| Train | 321,135 | 514.37 h | 129,885 / 227.72 h / 617 | 191,250 / 286.65 h / 473 |
| Val | 38,459 | 64.77 h | 16,531 / 28.93 h / 74 | 21,928 / 35.83 h / 55 |
| Test | 38,599 | 63.04 h | 15,495 / 27.21 h / 75 | 23,104 / 35.83 h / 55 |
train, val, and test.cross_split_overlap_count: 55,497). Evaluation separately reports Overlapped vs. Unseen prompt subsets.NVIDIA A100-SXM4-80GB (79.3 GB VRAM)2h 08m 54s (1.0 epoch, 5,020 steps)per_device_train_batch_size=16, gradient_accumulation_steps=4)bf16-mixed, tf32=true, gradient_clip_val=1.0, configured learning_rate=1e-4, min_lr=1.0e-06, warmup_steps=350, weight_decay=0.01, seed=42monitor=val_wer, mode=min, early_stopping_patience=4, eval_steps=400, save_steps=400LoraLayer modules via merge(safe_merge=True) + unwrapped back to base nn.Linear layers in float32, exported as .nemo (de9f88485a4b3960cbf1a77656072b70adb0eec19a9329df94aabccb0cf20831) and mapped identically to HF Nemotron3_5AsrForRNNT model.safetensors (37f0adbd08fed3884d9334ee1cac7404d6753fe759399ae04d281e5730386abc, max_diff = 0.0).metadata/training_config.yaml and metadata/export_manifest.json.All Korean evaluations use the identical single-source normalizer (shared_controller.normalize_ko) and Levenshtein CER/WER scorer with 1,000-resample Bootstrap 95% confidence intervals (att_context_size=[56, 13], target_lang="ko-KR"). Benchmark numbers below were measured on the combine LoRA weights prior to lossless merge; merged weights match W_base + (alpha/r) * (B @ A) with max_diff < 1e-6.
N = 38,599)| Condition | Overall CER (95% CI) | Overall WER | Overlapped Prompts (N=34,105) CER |
Unseen Prompts (N=4,494) CER |
|---|---|---|---|---|
Base (nvidia/nemotron-3.5-asr-streaming-0.6b) |
7.11% [0.0703 ~ 0.0720] |
24.01% | 7.02% | 7.60% |
| Combine LoRA / Merged (This Model) | 6.82% [0.0674 ~ 0.0691] |
23.75% | 6.71% | 7.40% |
| Evaluation Set | N | Base CER / WER | Combine CER / WER | Notes |
|---|---|---|---|---|
| Dysarthria Test (All) | 15,495 | 7.42% / 24.75% | 7.22% (7.24%*) / 24.25% | Slight improvement over base |
↳ Laryngeal (후두장애) |
11,931 | 6.05% / — | 5.89% / — | |
↳ Speech & Hearing (언어청각장애) |
1,269 | 15.77% / — | 15.19% / — | |
↳ Neurological (뇌신경장애) |
2,295 | 9.10% / — | 9.03% / — | |
| Typical Test | 23,104 | 6.92% / 23.57% | 6.57% (6.55%*) / 23.42% | Slight improvement over base |
External Korean Lecture (external_aihub71627) |
10,000 | 14.02% / 35.00% | 14.08% / 34.97% | Comparable to base |
*Note: 7.22% / 6.57% are from the 03_eval_finetuned.ipynb evaluation log; 7.24% / 6.55% are from the harmonized figure table (asr_visualization_scores.csv).
min_lr = 1.0e-06 after warmup, resulting in minimal weight divergence from the base checkpoint (7.11% -> 6.82% CER overall). For stronger dysarthric adaptation with Nemotron-3.5-ASR, full fine-tuning (dysarthria Full FT reached 3.92% CER) or tuning the NeMo scheduler configuration is recommended.N=2,847, separate inference protocol, descriptive reference only): 43.47% CER / 54.91% WER (vs. legacy Base 43.23% / 55.07%).nvidia/nemotron-3.5-asr-streaming-0.6b under OpenMDW-1.1; retain OpenMDW-1.1 notices on redistribution and comply with AI-Hub 013 & AI-Hub 109 data usage terms.