Cabin offline: two-way spoken LID (VoxLingua107 ECAPA, int8 ONNX) + fusion params
Language ID for the two-way (interpreting) mode of the Cabin offline app: which language of the session pair was
spoken, so only that recognizer has to run. The acoustic model is
speechbrain/lang-id-voxlingua107-ecapa (107 languages),
exported to ONNX and quantised (dynamic int8, per-channel QUInt8 weights); lid_fusion_params.json holds the
temperatures, thresholds and the per-pair logistic fusion with the app's ASR outputs, all fitted on a dev set that is
disjoint from the evaluation sets below.
- Status: candidate (host-verified; Android on-device benchmark pending)
- Collection: Cabin offline ASR (on-device, distilled from R13x)
- Code:
cabin-offline-asrmobile-local/lid/prod/(fit / eval / golden vectors); app modulecabin-mobile-appsrc/offline/spokenLid/(TypeScript: fbank, decision, scheduler)
Files
| path | what |
|---|---|
ecapa.int8pc.onnx |
input fbank float32 (1, T, 60), output logp float32 (1, 107) log-probabilities (order = labels.txt); sentence mean normalisation is inside the graph. 51 ms per 1.5 s window on one x86 core |
labels.txt |
the 107 VoxLingua107 codes (vi: Vietnamese, ...). No Cantonese: score yue with the zh column |
lid_fusion_params.json |
global: T_early 2.9, tau_early 0.73 (1.5 s), T_final 3.15, tau_final 0.71 (whole utterance); pairs: per app pair the fusion columns, StandardScaler mean / scale and logistic coefficients |
Every file is listed with its sha256 in SHA256SUMS.txt and with its size in FILES.tsv.
Front end (must match exactly)
speechbrain Fbank(n_mels=60): 16 kHz mono float in [-1, 1]; torch.stft n_fft 400, hop 160, periodic Hamming(400),
center=True with zero padding; power spectrum; 60 triangular mel filters 0-8 kHz (speechbrain's construction);
10*log10(max(x, 1e-10)), floored at (max over the input) - 80 dB. No deltas, no normalisation outside the graph. A
TypeScript port (fbank.ts) matches torch to 2e-3 dB; note that the int8 graph amplifies such tiny input differences
(max |Δ log-prob| ~0.5 on 24 clips, decisions unchanged on the held-out sets: 95.5 vs 95.3 % with the TS front end).
Decision (two-way session, languages a-b)
- 1.5 s after VAD onset (utterances >= 1.75 s): P(a) = sigmoid((logp[a] - logp[b]) / T_early); if max(P) >= tau_early that language is the tentative one (partials, which recognizer decodes first). No session prior, no hysteresis.
- VAD endpoint: whole-utterance (<= 30 s) P(a) with T_final; if max(P) >= tau_final it is final. Otherwise run both
recognizers and use the pair's logistic fusion of [logp[a] - logp[b], text features of both outputs] (features:
mobile-local/lid/prod/textfeats.py, e.g. valid-Vietnamese-syllable share, SenseVoice tag, function-word vote). - Pairs without fusion params: today's text rules. id-ms and th-lo: no automatic LID.
Results (held out; 95 % cluster bootstrap)
Interpreted interviews (private eval, 27 clips, 528 VAD segments, 170 language switches; vi-en 15 / vi-ko 12 clips):
| system | acc % | switch utterances % | vi-en % | vi-ko % | 2nd ASR run % |
|---|---|---|---|---|---|
| today's app rule (both ASRs + text rules) | 85.8 [80.0, 90.8] | 85.9 [80.6, 91.4] | 93.0 | 77.4 | 100 |
| this package (dev-fitted cascade) | 95.3 [93.1, 97.3] | 95.3 [92.1, 98.4] | 94.4 | 96.3 | 13.8 |
Multi-pair research eval (conversational YouTube + Common Voice, class-balanced accuracy %):
| pair | ECAPA argmax | app rule | cascade | 2nd ASR % |
|---|---|---|---|---|
| vi-en | 98.5 | 97.8 | 99.0 | 2.9 |
| zh-en | 98.7 | 99.2 | 99.0 | 4.1 |
| ja-en | 99.3 | 99.0 | 99.2 | 3.9 |
| ko-en | 99.1 | 99.0 | 98.8 | 4.6 |
| fr-en | 97.7 | 96.7 | 99.0 | 7.3 |
| de-en | 96.6 | 97.3 | 98.4 | 9.8 |
| es-en | 97.3 | 97.1 | 98.1 | 5.4 |
| zh-vi | 98.9 | 89.3 | 99.4 | 2.4 |
| ja-vi | 100.0 | 99.0 | 99.7 | 1.2 |
| ko-vi | 99.6 | 97.4 | 99.6 | 1.9 |
| mean | 98.6 | 97.2 | 99.0 | 4.4 |
Known weak spots: utterances < 1 s ("Good.", backchannels), disfluent Korean, VAD segments that hold two speakers in two languages (one label per segment), third-language turns (forced into the pair).
Fitting data and licences
- Model: Apache-2.0 (speechbrain/lang-id-voxlingua107-ecapa; HF API license:apache-2.0, checked 2026-10-29). Trained by SpeechBrain on VoxLingua107 (Valk & Alumäe 2021, CC BY 4.0). The ONNX export / quantisation is ours, same licence.
lid_fusion_params.jsonwas fitted on 3,200 clips (8 languages x 400): FLEURS (CC BY 4.0), Common Voice 17 (CC0), YODAS2 (CC BY 3.0) and R11c code-switch YouTube clips; every clip's video / speaker / transcript is disjoint from the eval sets above. No audio is redistributed here.
How to use (Python)
# pip install onnxruntime speechbrain soundfile huggingface_hub
import json, numpy as np, onnxruntime as ort, soundfile as sf, torch
from huggingface_hub import snapshot_download
from speechbrain.lobes.features import Fbank
d = snapshot_download("doanhlt/cabin-lid-ecapa-voxlingua107-onnx")
s = ort.InferenceSession(f"{d}/ecapa.int8pc.onnx", providers=["CPUExecutionProvider"])
codes = [l.split(":")[0] for l in open(f"{d}/labels.txt").read().splitlines()]
P = json.load(open(f"{d}/lid_fusion_params.json"))
x, sr = sf.read("utt16k.wav", dtype="float32") # 16 kHz mono
f = Fbank(n_mels=60)(torch.from_numpy(x)[None]).numpy()
lp = s.run(None, {"fbank": f})[0][0]
a, b = "vi", "en"
pa = 1 / (1 + np.exp(-(lp[codes.index(a)] - lp[codes.index(b)]) / P["global"]["T_final"]))
print(a if pa >= .5 else b, max(pa, 1 - pa), "confident" if max(pa, 1 - pa) >= P["global"]["tau_final"] else "-> fusion")