vits-melo-tts-zh_en-8k — MeloTTS zh_en with a distilled 8 kHz vocoder

Drop-in replacement for the sherpa-onnx vits-melo-tts-zh_en model directory, with the 44.1 kHz HiFi-GAN decoder replaced by a distilled 8 kHz Vocos-style vocoder (ConvNeXt @ 125 Hz + iSTFT head, 3.65 M params).

Built for a phone-attendant product on Jetson Nano gen1 CPU (4× Cortex-A57, ONNX Runtime CPU): the audio leaves through an 8 kHz G.711 channel, so rendering 44.1 kHz wasted ≥80 % of vocoder compute. Voice, prosody, and text front end are unchanged — only the decoder is new, distilled from MeloTTS's own output (teacher latent z → 8 kHz waveform). Identical input text produces the same speech, just 8 kHz and cheap.

Results

fp32, ONNX opset 17, no custom ops — runs on the sherpa-onnx-pinned ORT (~1.17) CPU build. Do not int8-quantize the convs: measured 9× slower than fp32 on the A57.

Listen — zh/en code-mixed samples

Same text, three renders: the stock teacher at its native 44.1 kHz, the teacher downsampled to 8 kHz (the theoretical best this model could sound), and this model rendered end-to-end by the stock sherpa-onnx engine. Small duration differences between teacher and student come from the sherpa C++ frontend vs melo's python frontend (tokenization), not from the vocoder.

1.「您好,這裡是全球科技,幫您轉接給 Kevin 陳經理,請稍候。」

2.「Amy 林目前正在開會,您的 Wi-Fi 報修單已經建立,編號五三三。」

3.「請撥打零二、二七一八、三三四四,分機二一八,聯絡 Jason 王工程師,謝謝您的來電。」

Note: the sherpa frontend drops raw digit tokens as OOV (same as the stock model dir), so digits in these samples are pre-normalized to Chinese (五三三). The deployed app already normalizes numbers before TTS; see INTEGRATION.md.

More material in ear_test/ (teacher/student pairs + G.711 μ-law channel renders).

Usage — stock sherpa-onnx, nothing changes but the model dir

sherpa-onnx-offline-tts \
  --vits-model=vits-melo-tts-zh_en-8k/model.onnx \
  --vits-tokens=vits-melo-tts-zh_en-8k/tokens.txt \
  --vits-lexicon=vits-melo-tts-zh_en-8k/lexicon.txt \
  --vits-dict-dir=<jieba dict dir, same as the stock model> \
  --sid=0 --num-threads=4 --output-filename=out.wav \
  "幫您轉接給 Kevin 陳經理,他的分機是 533。"

Output is 8 kHz mono (sample_rate=8000 in the model metadata).

Speaker id: use --sid 0 under sherpa-onnx (it's the Chinese voice)

This is a single-speaker export (n_speakers = 1 in the ONNX metadata), so the only in-range id is sid = 0, and that lone embedding is the Chinese (ZH) voice. Under stock sherpa-onnx, just pass --sid 0 (or omit --sid).

Why you may have seen "use sid 1". Upstream MeloTTS defines spk2id = {'ZH': 1}, so in the original model ZH is speaker 1. When this model was exported as a single-speaker ONNX, that ZH embedding became the only row — index 0 — and the original id is preserved only in the speaker_id metadata field. So they are the same voice under two numbering schemes:

If you pass --sid 1 to stock sherpa-onnx you'll get the warning sid should be in the range [0, 0]. Given: 1. Use sid=0; sherpa then clamps to 0, so the audio is still correct — but --sid 0 is the clean, warning-free call. (Verified: sid=0 and sid=1 produce byte-for-byte the same Chinese audio here.)

⚠️ Other GGUF/runtime ports number speakers differently. Some ports (e.g. RapidSpeech.cpp) take the speaker from the original spk2id space rather than the single-row ONNX index. For those, select the ZH speaker explicitly — in RapidSpeech.cpp use --lang Chinese, which maps ZH → speaker 1. Picking the wrong/default speaker there feeds the wrong embedding g into the flow/vocoder and degrades output. Bottom line: sherpa-onnx → sid 0; original-MeloTTS numbering → speaker 1; both refer to the same single Chinese voice.

Contents

model.onnx            melo enc/flow + distilled Vocos8k decoder (127 MB, fp32, opset 17)
tokens.txt            identical to the stock vits-melo-tts-zh_en export
lexicon.txt           identical to the stock vits-melo-tts-zh_en export
ear_test/             8 kHz WAVs: teacher vs student, clean + G.711, zh/en code-mixed lines,
                      plus renders produced by the stock sherpa-onnx engine
DEVICE_ACCEPTANCE.md  acceptance checklist: exact CLI, RTF bounds @1/2/4 threads, metrics
INTEGRATION.md        install steps + zh-TW (Traditional Chinese) input notes

zh-TW note

Traditional Chinese input works (the frontend reads it directly). The accent is Mainland Putonghua, not Taiwan Guoyu, and occasional polyphone errors exist — both inherited from the unchanged MeloTTS front end, identical to the stock 44.1 kHz model. See INTEGRATION.md.

Training / reproduction

Full training code, corpus generator, export and eval tooling: https://github.com/vieenrose/jetson-tts

Distillation recipe: dump (z, g, wav) from MeloTTS zh_en with bert=0 (matching the deployed sherpa export, which never computes BERT), downsample targets to 8 kHz, train a Vocos-style head (multi-res STFT + telephony-band mel + MPD/MSD GAN losses) on the latent→8 kHz mapping, then graft the trained decoder back into the melo ONNX graph.

Attribution

• Teacher: MeloTTS (MIT, MyShell.ai) — myshell-ai/MeloTTS-Chinese
• Export contract: k2-fsa/sherpa-onnx (Apache-2.0)
• Decoder architecture inspired by Vocos (MIT)

model.opset16.onnx — onnxruntime ≤ 1.11.0 / Jetson Nano gen1 variant

model.onnx is exported at ai.onnx opset 17 (uses LayerNormalization). onnxruntime 1.11.0 — the last version that supports the Jetson Nano gen1's CUDA 10.2 — caps at opset 16 and cannot load it (and onnx.version_converter can't downgrade LayerNormalization).

model.opset16.onnx is the same network with all 72 LayerNormalization nodes decomposed into opset-16 primitives (ReduceMean / Sub / Mul / Add / Sqrt / Div) and stamped at opset 16. It is numerically identical to model.onnx (max abs diff 9.2e-6, corr 1.000000) and loads + runs on onnxruntime 1.11.0 (validated on a cuDNN-free CUDA-10.2 build, ~45 ms / ~1 GB). Use it when you need to run melo8k on onnxruntime ≤ 1.11 (e.g. Jetson Nano gen1); use model.onnx on modern onnxruntime.

Reproduce: python melo8k_layernorm_to_opset16.py model.onnx model.opset16.onnx (script: github.com/vieenrose/edge-speech-gpu-bench tools/melo8k_layernorm_to_opset16.py).