cabin-asr-vi-zipformer-offline-r3 (release candidate rc1)
Vietnamese offline (non-streaming) zipformer2 transducer, 68.6 M parameters, BPE-2000, for the Cabin Offline app. Same architecture and file names as the shipped sherpa-onnx-zipformer-vi-int8-2025-04-20. Fine-tuned from r2 on 258 h (168 h new YODAS v1 Vietnamese transcribed by R13x, Apache-2.0, plus every r2 cut).
- Status: release candidate
rc1(2026-10-30): drop-in replacement for the shipped Vietnamese model, round 3 of the R13x → sherpa-vi distillation. Passed the host release check below (same as r2 rc1). Still unmeasured: real phone captures (Mac only), the app's invented-card suite and on-device (ARM) RTF; those gate the switch to shipped. - Collection: Cabin offline ASR (on-device, distilled from R13x). Repo names:
doanhlt/cabin-asr-<lang>-<arch>-<variant>-<version>. - Code:
thinkprompt/cabin-offline-asr(private GitHub repo), branchfeat/mobile-local-kit,mobile-local/distill/+mobile-local/README.md("Pilot: R13x → sherpa-vi distillation"); round 3 =distill/run_xq_r3.sh(cab_xq queue,distill/xq_runner.sh), gatedistill/gate_r3.py, release checkeval/r3rel_run.sh+eval/r2pkg_eval.py(RELCHK=r3).
Files
| path | what |
|---|---|
package/ |
the sherpa-onnx package the app loads (encoder / joiner int8, decoder fp32, tokens.txt, bpe.model) |
icefall/pretrained.pt |
icefall state dict (epoch 2, avg 1): start point for the next fine-tune (--finetune-ckpt) |
eval/ |
paired-bootstrap comparisons (vs shipped, r2, r3ctl control), round-3 gate, publish gate, dev-pick logs, mix stats |
data/yodas2_vi_heldout_*_clean.jsonl |
the two clean conversational held-out sets (refs) |
attribution/ |
YouTube video ids of all YODAS training audio (CC BY 3.0) |
package-fp32/ |
fp32 export of the same weights: reference for the int8 check, not for the phone |
eval/release_check/ |
rc1 release check: int8 vs fp32 vs shipped vs r2, thread sweep, spot checks, CPU RTF, interview ASR |
Every file is listed with its sha256 in SHA256SUMS.txt (sha256sum -c SHA256SUMS.txt) and with its size in FILES.tsv.
Results (WER %, lower is better; 95 % paired bootstrap over utterances, 10,000 resamples; * = CI excludes 0)
| set (n) | shipped | r2 | this | this − shipped | this − r2 |
|---|---|---|---|---|---|
| FLEURS-100 vi (gate <= 10.22) (100) | 10.19 | 7.44 | 7.64 | -2.55 [-4.47, -1.12] * | +0.20 [-0.26, +0.68] n.s. |
| CV13-vi test200 (200) | 9.58 | 8.37 | 8.37 | -1.21 [-2.47, +0.06] n.s. | +0.00 [-0.98, +0.98] n.s. |
| YODAS2 held-out clean, ref = R13x ∩ caption (463) | 7.43 | 5.61 | 5.12 | -2.31 [-2.73, -1.90] * | -0.49 [-0.79, -0.21] * |
| YODAS2 held-out clean, ref = caption (899) | 52.59 | 51.78 | 51.62 | -0.97 [-1.31, -0.64] * | -0.16 [-0.36, +0.04] n.s. |
Round-3 gate (pre-registered before training, distill/gate_r3.py → eval/gate_r3.json): fleurs_le_10.22 = pass, not_worse_than_r2 = pass, beats_r2_conversational = pass, beats_control_conversational = pass, beats_shipped_caption_clean = pass, size_within_2pct_of_r2 = pass.
Publish gate (eval/gate_publish.json): fleurs100_le_10.22 = pass, sig_better_than_shipped_on_ge_2_sets = pass, not_sig_worse_than_previous_on_any_set = pass, size_within_5pct_of_shipped = pass, rtf_within_10pct_of_shipped = pass. Package 79.3 MB vs shipped 77.4 MB (same graph).
Control arm (r3ctl: same start point, epochs, LR and seed on the r2 cuts only, i.e. "three more epochs on the old data"):
| set | r3ctl | r3 | r3 − r3ctl |
|---|---|---|---|
| FLEURS-100 vi (gate <= 10.22) | 7.61 | 7.64 | +0.03 [-0.44, +0.49] n.s. |
| CV13-vi test200 | 8.23 | 8.37 | +0.13 [-0.86, +1.13] n.s. |
| YODAS2 held-out clean, ref = R13x ∩ caption | 5.56 | 5.12 | -0.43 [-0.69, -0.19] * |
| YODAS2 held-out clean, ref = caption | 51.68 | 51.62 | -0.06 [-0.25, +0.13] n.s. |
Honest reading: against r2 the gain is on the conversational YouTube held-out set with the R13x ∩ caption reference (−0.49, CI excludes 0) and it comes from the new data (the control does not move it); FLEURS, CV13 and caption-clean are within noise of r2. Against the shipped model r3 is significantly better on FLEURS-100, agree-clean and caption-clean (CV13 −1.21, CI touches 0). Not measured on real phone captures, the app's invented-card suite or ARM RTF yet: those gate a release candidate.
Release check (rc1, 2026-10-30)
The package/ files are unchanged since the candidate upload (sha256 equal; re-exporting icefall/pretrained.pt with icefall
export-onnx.py gives byte-identical ONNX files). Built exactly like the app builds its Vietnamese recognizer:
OfflineRecognizer.from_transducer, greedy search, 16 kHz, 80-dim fbank, encoder / joiner int8 (QInt8, shipped scheme),
decoder fp32. package-fp32/ = fp32 export of the same weights (reference only). Host: AMD EPYC 7313 16-Core Processor.
| set (n) | shipped int8 | r3 fp32 | r3 int8 | int8 − fp32 | int8 − shipped |
|---|---|---|---|---|---|
| FLEURS-100 vi (gate <= 10.22) (100) | 10.19 | 7.61 | 7.64 | +0.03 [-0.11, +0.18] n.s. | -2.55 [-4.47, -1.12] * |
| CV13-vi test200 (200) | 9.58 | 8.37 | 8.37 | +0.00 [-0.46, +0.41] n.s. | -1.21 [-2.47, +0.06] n.s. |
| YODAS2 held-out clean, ref = R13x ∩ caption (463) | 7.43 | 5.16 | 5.13 | -0.03 [-0.13, +0.07] n.s. | -2.31 [-2.73, -1.89] * |
| YODAS2 held-out clean, ref = caption (899) | 52.58 | 51.59 | 51.62 | +0.03 [-0.06, +0.12] n.s. | -0.96 [-1.30, -0.63] * |
Table at 2 threads. Thread-stable: r3 int8 gives the same WER at 1, 2 and 4 intra-op threads (agree-clean 5.13 / 5.13 / 5.12); r2's FLEURS flip between 1–2 and 4 threads (7.98 / 7.44) is gone. int8 − fp32 pooled over FLEURS + CV13 + agree-clean: -0.02 [-0.11, +0.07] n.s. (+0.30 budget holds on every set and thread count).
r3 int8 − r2 int8 (both decoded on this host at the same thread count):
| threads | fleurs100_vi | cv13_vi_test200 | yodas2_vi_heldout_agree_clean | yodas2_vi_heldout_caption_clean | pooled FLEURS + CV13 + agree |
|---|---|---|---|---|---|
| 1 | 7.98 → 7.64 : -0.34 [-1.55, +0.50] n.s. | 8.37 → 8.37 : +0.00 [-0.98, +0.98] n.s. | 5.61 → 5.13 : -0.48 [-0.78, -0.20] * | 51.76 → 51.62 : -0.14 [-0.34, +0.06] n.s. | -0.41 [-0.73, -0.13] * |
| 2 | 7.98 → 7.64 : -0.34 [-1.55, +0.50] n.s. | 8.37 → 8.37 : +0.00 [-0.98, +0.98] n.s. | 5.61 → 5.13 : -0.48 [-0.78, -0.20] * | 51.76 → 51.62 : -0.14 [-0.34, +0.06] n.s. | -0.41 [-0.73, -0.13] * |
| 4 | 7.44 → 7.64 : +0.20 [-0.26, +0.68] n.s. | 8.37 → 8.37 : +0.00 [-0.98, +0.98] n.s. | 5.61 → 5.12 : -0.49 [-0.79, -0.21] * | 51.78 → 51.62 : -0.16 [-0.36, +0.04] n.s. | -0.32 [-0.57, -0.08] * |
Spot checks (numbers, English loanwords / foreign names = tokens that are not a well-formed Vietnamese syllable, short clips):
| subset (pooled FLEURS-100 + CV13 + agree-clean) | n | shipped | r3 int8 | int8 − shipped | empty output ship / r3 |
|---|---|---|---|---|---|
| ref_has_digits | 13 | 25.42 | 21.34 | -4.08 [-7.56, -1.14] * | 0 / 0 |
| ref_has_foreign_word | 61 | 17.36 | 13.13 | -4.23 [-7.57, -1.71] * | 0 / 0 |
| short_le_2s | 8 | 9.68 | 9.68 | +0.00 [+0.00, +0.00] n.s. | 0 / 0 |
| short_le_3_words | 13 | 20.69 | 24.14 | +3.45 [-15.38, +19.23] n.s. | 0 / 0 |
Long deletions (utterances with ≥ 6 consecutive reference words dropped, FLEURS + CV13 + agree-clean): shipped 2, r2 1, r3 0. Empty outputs over 1,662 clips: shipped 44, r2 9, r3 56 (all on caption-clean, mostly ≤ 2 s clips whose caption does not match the audio; on the 45 short caption clips with a plausible reference: shipped 8, r2 6, r3 14; on real interview units 1 / 583).
Real interpreter interviews (eval/interviews, 583 Vietnamese VAD units / 57 videos, R13x reference, video-cluster bootstrap): shipped 16.35 → r3 14.11 (-2.24 [-2.87, -1.68]); r3 − r2 -0.01 [-0.24, +0.21] (tie); Gemini reference r3 − shipped -0.72 [-2.08, +0.19]; ko-vi sessions r3 − r2 +0.33 [+0.06, +0.71].
Output format = shipped: same tokens.txt / bpe.model, upper case, no punctuation, numbers as words (all upper:
True, outputs with digits 0, with punctuation 0).
CPU real-time factor (interleaved, median of 3 rounds, 120 clips / 1021.0 s; x86, not a phone):
| threads | shipped int8 | r2 int8 | r3 int8 | r3 fp32 | median latency clips ≤ 2 s, ms (ship / r2 / r3) |
|---|---|---|---|---|---|
| 1 | 0.0233 | 0.0233 | 0.0233 | 0.0318 | 43 / 42 / 42 |
| 4 | 0.0190 | 0.0189 | 0.0185 | 0.0221 | 36 / 37 / 37 |
Size: package/ 79.3 MB vs shipped 77.4 MB (+2.5 %); fp32 272.7 MB.
Full report: eval/release_check/REPORT.md. Pin downloads to the rc1 tag (revision="rc1").
Training data, licences and attribution
Total 257.82 h (172,949 cuts), 17 % human-labelled: every round-2 cut (31,602 of 32,180 after an eval-overlap guard) plus 168.25 h of new YODAS (v1) Vietnamese.
| part | hours | labels | licence |
|---|---|---|---|
| FLEURS vi train replay x3 | 10.289 | human | CC BY 4.0 |
| r1 YODAS2 vi000 | 26.401 | R13x (judge vs a second label / caption) | CC BY 3.0 |
| r2 CV13 vi train | 3.286 | human | CC0 |
| r2 FOSD | 30.802 | human | CC BY 4.0 |
| r2 R11c code-switch YODAS | 18.797 | R13x (judge vs a second label / caption) | CC BY 3.0 |
| r3 CV17 vi, not in CV13 | 0.054 | human | CC0 |
| r3 YODAS v1 R12c vi_yodas_extra re-filtered | 7.605 | R13x (judge vs a second label / caption) | CC BY 3.0 |
| r3 YODAS v1 S8 clean pool | 160.59 | R13x (judge vs a second label / caption) | CC BY 3.0 |
New round-3 rows: the YODAS v1 vi000 / vi100 S8 clean pool and the R12c vi_yodas_extra rows, relabelled by R13x and
kept only where R13x agrees with the earlier label (interior WER ≤ 0.3, plus digit / length / language / short-utterance
filters); any row sharing exact normalised text or an 8-gram with a local eval transcript was dropped.
Attribution:
- YODAS / YODAS2 (Li et al., ASRU 2023; espnet): CC BY 3.0. YouTube video ids of every training clip:
attribution/yodas2_video_ids.txt(pilot1 rows),attribution/yodas_r11c_video_ids.txt(code-switch rows),attribution/yodas_v1_r3_video_ids.txt(round-3 rows). - FPT Open Speech Dataset (FOSD), © FPT Corporation, CC BY 4.0. Common Voice 13 / 17 vi: CC0 (CV13 test200 sentences dropped).
- FLEURS (Google): CC BY 4.0. MUSAN (CC BY 4.0) / ESC-10 (CC BY) as additive noise only.
Licence: Apache-2.0 (base zzasdf/viet_iter3_pseudo_label Apache-2.0; teacher R13x, a Qwen3-ASR-1.7B fine-tune,
Apache-2.0). No Gemini, Soniox or OpenAI output is used (labels, filters or text), no user capture, no eval set: FLEURS
test/dev, CV13 test200, the YODAS2 dev / held-out videos and the interpreter-interview test videos are excluded from training.
Recipe
- icefall zipformer
finetune.py --do-finetune 1from r2'spretrained.pt; LR 0.0008 (Eden lr_epochs 100, lr_batches 100k), 3 epochs, max-duration 200, fp16, one shared A100 (10 GB cap). - Picked on dev only (FLEURS-vi dev + YODAS2-vi dev, mean WER) among (epoch, avg) 3:1 3:2 2:1: epoch 2, avg 1.
- Export: icefall
export-onnx.py, encoder / joiner int8, decoder fp32, non-causal (same as r2 / shipped).
How to use (sherpa-onnx, Python)
# pip install sherpa-onnx soundfile huggingface_hub
import sherpa_onnx, soundfile as sf
from huggingface_hub import snapshot_download
d = snapshot_download("doanhlt/cabin-asr-vi-zipformer-offline-r3", allow_patterns=["package/*"], revision="rc1") + "/package"
rec = sherpa_onnx.OfflineRecognizer.from_transducer(
encoder=f"{d}/encoder-epoch-12-avg-8.int8.onnx", decoder=f"{d}/decoder-epoch-12-avg-8.onnx",
joiner=f"{d}/joiner-epoch-12-avg-8.int8.onnx", tokens=f"{d}/tokens.txt",
num_threads=4, decoding_method="greedy_search") # greedy = what the app runs
audio, sr = sf.read("clip.wav", dtype="float32") # mono; sherpa-onnx resamples to 16 kHz
s = rec.create_stream(); s.accept_waveform(sr, audio); rec.decode_stream(s)
print(s.result.text)
The file names are those of the shipped sherpa-onnx-zipformer-vi-int8-2025-04-20, so the app swaps the model by
directory only.