Thaki-ASR-KO-Dialects-v1

Qwen3-ASR-1.7B fine-tuned on five Korean regional dialects (Gangwon, Gyeongsang, Jeolla, Jeju, Chungcheong), with standard Korean kept intact. Pooled character error rate on sealed dialect test sets drops from 13.52% to 10.81% (relative −20.1%); every region improves; Zeroth-Korean (standard read speech) moves from 3.83% to 3.66%.

🇰🇷 한국어 카드: MODEL_CARD.ko.md

⚠️ This is a research-gate release, not a production-validated model. It was evaluated on clean, single-channel AI Hub recordings with a female-skewed speaker pool. Noisy or far-field audio, male speakers, children, and non-Korean languages were not evaluated (§5).

0. Quick start

The checkpoint is a drop-in replacement for Qwen/Qwen3-ASR-1.7B. Use the official qwen-asr package.

pip install -U qwen-asr

import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "ThakiCloud/Thaki-ASR-KO-Dialects-v1",
    dtype=torch.bfloat16,
    device_map="cuda:0",
    max_inference_batch_size=16,
    max_new_tokens=256,
)

# audio: a local path / URL to a 16 kHz mono WAV, or a (numpy array, sample_rate) tuple
results = model.transcribe(audio="jeju_sample.wav", language="Korean")
print(results[0].text)

Decoding used for every number on this card: greedy (do_sample=False), language="Korean" forced, max_new_tokens=256, bf16 inference on one H100. The model also accepts the base model's other 51 languages, but we did not evaluate them (§5).

Transcription conventions follow the training labels: AI Hub dialect transcripts (spoken form, no punctuation normalisation) and Zeroth-Korean (numbers written out in Korean, e.g. 이천 십 일 년). Expect Korean-numeral output more often than the base model.

1. What changed, and what did not

Numbers below are our own measurements of the base model and this checkpoint, same sealed sets, same decoding, same hardware (1× H100 NVL, dev cluster), 2 repetitions each (greedy decoding is deterministic here: both repetitions produced identical character-level output, so no spread is shown).

1.1 Dialect test sets (sealed before training)

How the pooled number is computed. The pre-registered gate (GD1) defines the aggregate CER as total substituted + deleted + inserted characters ÷ total reference characters across the five regional sets (3,425 / 25,332 for the base, 2,738 / 25,332 for v1). This weights regions by their reference-character count, not by utterance count, which is why it differs from the utterance-weighted mean of the regional CERs shown in the last row. Both are reported; the gate was judged on the pooled figure.

Jeolla improved the least (−11.1%); it is the only region whose single-region improvement would not have met a 15% bar on its own. The gate as registered asks for a pooled −15% and no regional regression, which v1 meets.

1.2 Dialectal vs. standard-form utterances

Each test set was split by whether the AI Hub transcript marks the utterance as containing dialectal forms (dialectal) or not (standard). This separates Jeju language from standard Korean spoken by Jeju residents. Splits with fewer than 20 utterances were not evaluated; splits under n = 150 are reported but were not gate criteria.

Read the small splits (Jeolla dialectal n = 27, Chungcheong dialectal n = 51, Gangwon dialectal n = 63) as direction only. One utterance in the Jeolla dialectal split is roughly 3.7 percentage points of its CER.

1.3 Standard Korean guardrail

The standard-Korean guardrail only says "did not get worse on read speech". It does not claim a general Korean ASR improvement.

1.4 Latency

Architecture and parameter count are unchanged from the base model, so throughput is the same. On the evaluation run (1× H100 NVL, bf16, batch 1): median latency 0.39–0.61 s per 3–8 s utterance, real-time factor 0.11–0.14.

2. Training data

• Dialect hours were capped at 60 h per region (Gangwon has less available). Within a region, whole recording sessions were sampled so that test speakers never appear in training.
• Leak filtering: 0 training utterances share audio with any test set; 4,750 training utterances whose transcript exactly matched a test reference were dropped. Final training set: 273,217 utterances; 200 held-out utterances for training-time eval loss.
• Speaker skew: 78–89% of dialect training utterances per region are from female speakers (Gangwon 81%, Gyeongsang 86%, Jeolla 89%, Jeju 78%, Chungcheong 83%), and speakers in their twenties dominate. The sealed test sets have the same skew (Chungcheong test is 100% female). Male and older-speaker performance is therefore under-measured.
• The Zeroth-Korean test split and the five sealed dialect test sets were never used for training or model selection.

3. Training recipe

An earlier run with the audio encoder frozen (same data, bf16) improved Jeju by only 9% relative, so for dialect adaptation the encoder update appears to matter. An earlier run with 10 h per region (full fine-tune) reached a pooled −11.2% and did not pass the gate; the 60 h cap did.

4. Evaluation protocol

• Pre-registered gates (frozen before training, with a self-test on earlier failed runs): GD1 pooled dialect CER relative change ≤ −15% and no region (n ≥ 150) worse than base; GD2 Zeroth test CER ≤ base + 0.3 pp. Verdict: both passed.
• Sealed sets: per region 300 utterances (Chungcheong 195) drawn from recording sessions excluded from training; Zeroth test 200 utterances sampled once and frozen.
• Scoring: character error rate after Unicode NFC normalisation, whitespace collapsing and punctuation removal; the same scorer was used for base and candidate. Numerals are not normalised in the headline numbers (a numeral-normalised CER is in the ledger and differs by < 0.1 pp).
• Two repetitions per set; greedy decoding produced identical outputs, so the measurement-noise band used by the gate was 0.

Raw per-utterance hypotheses and the gate verdict file are kept in our internal ledger; the numbers on this card are copied from it without rounding beyond two decimals.

5. Limitations and what we did not measure

• Not production-validated. Clean studio-style AI Hub audio only. No noise, far-field, telephone, or overlapping-speech evaluation.
• Female and young speakers dominate both training and test. Treat male, child and elderly-speaker accuracy as unknown.
• Other languages were not evaluated. The base model supports 52 languages. In a separate Korean-only fine-tune of the same base we observed Japanese CER nearly doubling, so do not assume multilingual ability is preserved here. Use the base model for non-Korean audio.
• Jeolla gains are modest (−11.1%) and the Jeolla dialectal split is tiny (n = 27).
• Jeju remains hard: 23.6% CER overall and 28.4% on dialectal utterances. This is a large improvement, not a solved problem.
• Transcription style follows AI Hub / Zeroth conventions (spoken forms, Korean numerals); downstream text normalisation may be needed.
• No timestamp / forced-alignment evaluation was done (the base model's aligner should still work, untested).

6. Data licensing and what is (not) in this repository

• This repository contains model weights and configuration files only. No AI Hub audio, transcripts, metadata, or derived data files are included, in line with the AI-Hub terms of use, which allow the data to be used for training AI models and prohibit providing the data, or simple edits of it, to unapproved third parties.
• Source attribution (required by the AI-Hub terms, which extend the requirement to derivative works): 이 모델은 한국지능정보사회진흥원(NIA)의 사업결과인 AI-Hub "한국어 방언 발화(강원도)", "한국어 방언 발화(경상도)", "한국어 방언 발화(전라도)", "한국어 방언 발화(제주도)", "한국어 방언 발화(충청도)" 데이터를 활용해 학습했습니다. (This model was trained using the AI-Hub datasets "한국어 방언 발화" for Gangwon, Gyeongsang, Jeolla, Jeju and Chungcheong, which are results of a project of the National Information Society Agency (NIA), Korea.)
• Zeroth-Korean is CC BY 4.0 (Lucas Jo, Wonkyum Lee; mirrored as Bingsu/zeroth-korean).
• Base model Qwen/Qwen3-ASR-1.7B is Apache-2.0. The fine-tuned weights are released under Apache-2.0.

7. Reproducibility

• Base checkpoint: Qwen/Qwen3-ASR-1.7B (Hugging Face, Apache-2.0).
• Training: official qwen-asr SFT recipe, hyperparameters in §3, transformers 4.57.6, torch 2.9.0+cu130, flash-attn 2.8.3.
• Evaluation: greedy decoding, language="Korean", scorer as in §4. The sealed dialect test sets are AI Hub-derived and cannot be redistributed; the Zeroth test sample can be reconstructed from Bingsu/zeroth-korean test split (first 200 utterances after our deterministic ordering are listed in the ledger on request).
• File integrity: SHA-256 of every uploaded file was verified against the training job's manifest before and after upload (see the upload commit message / ledger).

8. Citation

@misc{thakicloud2026thakiasrdialects,
  title  = {Thaki-ASR-KO-Dialects-v1: Qwen3-ASR-1.7B fine-tuned on five Korean regional dialects},
  author = {ThakiCloud},
  year   = {2026},
  url    = {https://huggingface.co/ThakiCloud/Thaki-ASR-KO-Dialects-v1}
}

Base model: Qwen Team, Qwen3-ASR Technical Report (arXiv:2601.21337).