Uzbek ASR — open-weight GigaAM-Multilingual CTC fine-tunes

Live API free 30 min: clarion.uz

Commercial use, on-prem deployment & custom fine-tuning: rustam@clarion.uz · startups: free program at https://clarion.uz/startups

Open-weight Uzbek speech recognition with punctuation, fine-tuned from GigaAM-Multilingual (MIT) on 1.6k hours of Uzbek speech, then adapted to spontaneous speech on another ~250 hours of cross-checked YouTube transcripts.

Open-weight, research license: the weights are downloadable (gated form) and free for evaluation and non-commercial research; commercial use is available on request under a written license.

This repository holds the full ablation ladder — eight checkpoints, each one a question that was asked and answered — not just the final weights. The corpus definition and the evaluation set are published alongside it, so every number below can be checked rather than taken on trust:

• uzbek-asr-train-manifests — the exact rows every run saw, with hashes
• uzbek-asr-benchmark-spontaneous — the human-verified test set, with per-clip predictions

Results

Two test sets, because one of them flatters everybody.

Spontaneous speech — the number that matters

311 human-verified clips of real Uzbek conversation (benchmark, v3 — references corrected by a second, listening-based review on 2026-10-28). Corpus-level WER/CER, text lowercased, apostrophes canonicalized, punctuation stripped on both sides.

† Re-run on 2026-10-28. The original August run gave 39.6% on v2; the 0.6-point gap comes from the runtime environment (same code, weights and references).

Models without kept per-clip predictions, scored on benchmark v2 only — compare them with the "WER on v2" column above, not with v3:

The canonical column joins Uzbek clitics on both sides before scoring — see below for why that is the fairer number. Per-clip predictions for the shipped configurations are in the benchmark repository, so the table can be recomputed line by line. On v3, the shipped model is 8.1% against 14.4% for untuned GigaAM-large and 38.6% for whisper-turbo-uzbek. The development notes below quote v2 numbers, the ones measured at the time.

KenLM decoding takes the same weights from 12.2% to 11.5% without training, and the 220M from 13.9% to 12.2%: the weaker the acoustic model, the more the language model recovers. A 4-gram word model over 387k punctuated training sentences (7.1M words) rescores 256 CTC beams; it mostly fixes one-or-two-letter spelling slips, the largest error category. Weights (alpha 0.2, beta 0) were chosen by 2-fold cross-validation on the benchmark.

Spontaneous-speech adaptation (2026-10) took the 600M from 11.5% to 10.8% in two more stages and a larger LM. large_domain_600m continued training on a spontaneous-heavy mix (LLM-transcribed YouTube, podcasts, IT talks) with ~19% read-speech replay. large_agree_600m then added ~250 hours of YouTube speech whose transcript was cross-checked: our own model's transcript and YouTube's auto-captions either agreed (≤15% word difference), or were reconciled by an LLM that saw both and was forbidden to add words in neither. Clips where the two disagreed completely (usually music or overlapping speakers, 123 h) were left out. Each stage moved greedy WER by about 0.2–0.5 points; the larger LM (below) added 0.25. Checkpoint averaging, a soup of two runs, a looser speaking-rate filter and adding the new transcripts to the LM text were all tried and moved nothing beyond ±0.05.

Correction (2026-10-17). Earlier versions of this card reported 10.2% / 10.8% with KenLM. That LM's text contained 62 of the 311 benchmark clips: the same YouTube recordings also appear in two other training sources (yt_it, yt_podcasts), which the row-level exclusion did not catch. On those clips the old LM scored 5.8% WER — it had memorised them. The LM is now decontaminated (any sentence sharing a 10-word sequence with a benchmark reference is dropped, with ±25 neighbouring rows), and the numbers above are the honest ones. On the 249 clean clips the new LM is slightly better than the old one (11.42% vs 11.53%). The acoustic models were checked the same way and are unaffected: greedy WER on the 62 clips is higher than on the rest.

Throughput. One RTX 4090, the 1.98-hour set, production code path: 600M greedy 163× realtime, 600M + KenLM 163× with the default beam 64 (114× for uzasr-large-v0.2.1 at beam 256, measured back to back — beam 256 and 64 give identical WER here), 220M greedy 193×, 220M + KenLM 168× with beam 64 (126× for uzasr-base-v0.2.1 at beam 256). VRAM: 3.6 GB for 600M, 1.4 GB for 220M. bf16 autocast changed neither speed nor accuracy at this request size.

FLEURS (read speech)

650 samples, 2.13 h. Included for comparability with published results, not because it measures the product.

Read the two tables together and the point of the second one becomes clear: on FLEURS the fine-tune looks like a rounding error, on real speech it is 1.6 points. The 600M checkpoints were never scored on FLEURS — by then the question had stopped being interesting.

Validation

Held-out, speaker-disjoint, punctuated. Same split for every run below, so these are the numbers the ablation was actually steered by.

Which checkpoint

The remaining checkpoints (base_220m, gemini_bench_220m, gemini_full_220m, base_domain_220m, large_bench_600m, large_full_600m, large_domain_600m) are ablation stages, kept so the ladder below is auditable. For large_agree_600m and base_agree_220m the published file is the last step, not best.pt: the 300-sentence validation picked an earlier step that is worse on the benchmark (10.74% vs 10.62% with the LM, logit protocol), so validation is not trusted to choose at this scale.

Language model. lm/uz-4gram-2026-10-27.bin (KenLM binary, 4-gram, pruned and 8-bit quantized, 248 MB), lm/uz-4gram-2026-10-27.unigrams.txt, and .json with the corpus hash and decontamination settings. 10.2M sentences: the training transcripts plus tahrirchi/uz-crawl (Apache-2.0), with every sentence sharing a 10-word sequence with a benchmark reference dropped. Decoding: beam 64; alpha 0.2, beta 0.0 for both sizes (2-fold cross-validation). The smaller lm/uz-4gram-2026-10-17.* is the LM of the v0.2.1 tags. The leaked uz-4gram-2026-10-16 LM is removed from main; the *-0.2.0 tags still contain it.

Release tags. Each production version is an immutable tag — pin it instead of main:

Usage

pip install torch transformers huggingface_hub soundfile numpy
wget https://huggingface.co/rustam1221/uzbek-asr-gigaam/raw/main/inference.py

python inference.py interview.mp3
python inference.py interview.mp3 --variant 220m --device cpu

pip install pyctcdecode kenlm          # optional, for --lm
python inference.py interview.mp3 --lm

from inference import UzbekAsr

asr = UzbekAsr.from_hub("large-600m", lm=True)
print(asr.transcribe("interview.mp3"))
# Assalomu alaykum, bugungi suhbatimizda ta'lim tizimi haqida gaplashamiz.

inference.py is standalone: it downloads the checkpoint, rebuilds the extended CTC head, restores the punctuation vocabulary, splits audio past the model's 30 s limit at its quietest point, and applies sentence-start capitalization. ffmpeg must be on PATH — any audio or video container works.

How it was built

Why GigaAM and not Whisper

Whisper is the obvious starting point and it was the wrong one for Uzbek. On spontaneous speech the best available Uzbek Whisper fine-tune (whisper-large-v3-turbo-uzbek) scores 39.6% WER, against 16.1% for the untuned GigaAM-large. Fine-tuning cannot close a gap that size; the base model choice decided the outcome before any training ran.

GigaAM-Multilingual is character-level CTC, which also made punctuation cheap — see below — and its MIT license allows the fine-tuned weights to be licensed commercially.

Punctuation as four extra CTC rows

Most Uzbek ASR output is an unpunctuated lowercase stream, which is close to unusable as a product. Instead of bolting on a separate punctuation-restoration model, . , ? ! were added directly to the CTC vocabulary: the head's output convolution gains four rows, initialized to zero weight and bias −8, so the new symbols start effectively silent and the warm-started model does not regress while it learns them.

Every punctuation run therefore reports two validation numbers: WER with symbols, and WER with symbols stripped. The second one is the regression guard, and it held — 8.26% before the punctuation phase, 8.03% after. The model learned to punctuate for free, from prosody, and got slightly better at words while doing it.

Sentence-start capitalization is then a deterministic rule over the punctuation (inference.py), not something the model has to spend capacity on.

The ablation ladder

Each run warm-starts from the one above it. Validation is the held-out speaker-disjoint split of the training corpus.

¹ not comparable with the rows below — that run had no punctuation in its targets. ² validation mix changed with the new sources; these two stages were judged on the benchmark only (greedy 11.93% → 11.46% → 11.25%, logit protocol). ³ the large recipe replayed on the 220M, judged on benchmark v3 only: greedy 12.34% → 11.32% → 11.05%, with KenLM 10.23% → 9.74% → 9.42% (logit protocol). The smaller model gains twice as much from the same two stages.

All runs: character-level CTC, bf16, SpecAugment (2 frequency masks × 8 bins, 2 time masks × 5%), AdamW with weight decay 0.01, gradient clipping at 1.0, cosine schedule after warmup, fixed seed, on a single RTX 4090 16 GB. Batches are measured in audio-seconds rather than utterances so step size stays constant across a corpus with 0.5–30 s clips: 480 audio-seconds per step for the 220M runs, 468 for the 600M run.

large_full_600m was stopped at step 6,000 of a planned 8,000. The published checkpoint is the best validation point reached; the last two thousand steps were never run, so the 600M line is a floor rather than a converged result.

What the ladder showed

FLEURS does not measure this task. After the base fine-tune, FLEURS said 9.4% against the baseline's 9.5% — no gain worth reporting. On spontaneous speech the same checkpoint went from 17.1% to 15.5%. FLEURS is read audiobook-style speech; if it had been the only benchmark, the entire training run would have looked like a waste. This is why the spontaneous benchmark was built before the training budget was spent.

Data returns saturate fast, and cheaply-measured. LLM-transcribed YouTube speech was the obvious way to buy in-domain data. Rather than train on all 660 hours and find out, a 120-hour capped run went first: it bought 1.0 WER point. The full 660 hours then bought roughly 0.1 more on the benchmark. Five and a half times the data, a twentieth of the gain — the cap experiment paid for itself several times over.

Capacity picked up where data stopped. With 220M saturated, 600M was the remaining lever, and it moved the number the data no longer could: 13.9% → 12.2% on spontaneous speech, 1.7 points, against the ~0.1 points the last 540 hours of data had bought. Note also that the 600M model reached 11.08% validation WER in 4,000 steps from the raw base model — better than the 220M ladder managed in 22,000. And it did that without finishing: the run stopped at 6,000 of 8,000 steps.

Part of the remaining gap was measurement, not recognition. Uzbek clitics (da, ku, chi, mi, …) are written both joined and separated in real text, and neither spelling is wrong. Scoring them as errors charged the model 0.6 WER points for an orthographic convention. Canonicalizing both sides before scoring is the honest measurement; it was worth checking, and it was worth knowing that it explained only 0.6 points rather than the 3–4 first estimated.

Training data

916 hours of training audio across seven public corpora — UzbekVoice, Uzbek Speech Corpus, Common Voice, FLEURS, and three YouTube collections (974 h including the validation and test splits) — plus roughly 660 hours of LLM-transcribed spontaneous YouTube speech. The adaptation stages add ~250 hours of cross-checked YouTube speech (above); those transcripts are kept private, like all YouTube-derived audio. Filtering, text normalization, speaker-disjoint splitting and the exact row-level manifests are documented in uzbek-asr-train-manifests.

The 315 benchmark rows are excluded from training and the exclusion was verified — zero leaked rows.

Limitations

• 10.8% is not yet at the target. The goal is 10% WER on spontaneous speech; 0.8 points remain (0.2 canonical). Decoding-side tricks are exhausted (see above); more in-domain audio and better handling of overlapping speech are the open levers.
• The error tail is overlapping speech. The worst 30 benchmark clips carry 24% of all errors, and they are 25–30 s multi-speaker segments where turns overlap. Half the substitution errors are within two characters of the reference — spoken-form spelling variants rather than misheard words.
• Proper nouns stay lowercase. Only sentence-start capitalization is applied; a named-entity layer is not implemented.
• Comma placement follows written grammar, inherited from the training transcripts, and does not always match the speaker's pauses.
• Code-switching into Russian and English is the weakest category after overlap: roughly 7% of substitutions.
• No streaming. Decoding runs over whole utterances (up to 30 s per chunk); a streaming decoder is open.

Repository layout

checkpoints/<run>/best.pt   weights, plus the run config and vocabulary
checkpoints/<run>/step_*.pt  a fixed-step checkpoint, same format
configs/<run>.yaml          the exact training config each run used
lm/uz-4gram-<date>.*        KenLM binary, unigrams and corpus metadata
inference.py                standalone transcription script
requirements.txt

Each best.pt carries the full training config, step count, best validation WER and character vocabulary, so a checkpoint is self-describing — the configs/ files are extracted from the checkpoints themselves, which is how the 600M configs survived the loss of the training machine.

License and provenance

Open-weight release under the Uzbek ASR Commercial License — see LICENSE.md. Free for evaluation and non-commercial research; any commercial use requires a written license agreement. Contact: rustam@clarion.uz · live demo: https://clarion.uz/demo

Base model: GigaAM-Multilingual by GigaChat Team (ai-sage), MIT License — its copyright notice is reproduced in LICENSE.md.

Citation

@misc{uzbek_asr_gigaam,
  title  = {Uzbek ASR: GigaAM-Multilingual CTC fine-tunes with punctuation},
  author = {Nuriddinov, Rustamjon},
  year   = {2026},
  url    = {https://huggingface.co/rustam1221/uzbek-asr-gigaam}
}