CosyVoice3 ONNX Models

ONNX-exported models for Fun-CosyVoice3-0.5B, enabling PyTorch-free inference.

Model Description

CosyVoice3 is a multilingual text-to-speech (TTS) system with zero-shot voice cloning capabilities. This repository contains ONNX-converted models for pure ONNX Runtime inference without PyTorch dependencies.

Supported Languages

CosyVoice3 automatically detects the language from text content. Supported languages include:

• English, Chinese, Japanese, Korean
• German, Spanish, French, Italian, Russian
• Cantonese and other Chinese dialects

Important: Do NOT use language tags like <|en|> or <|ja|> - they will be pronounced as literal text.

Repository Contents

This repository includes everything needed for inference (no additional downloads required):

.
├── *.onnx                              # ONNX model files (14 files, ~3.8GB total)
├── vocab.json                          # Qwen2 tokenizer vocabulary
├── merges.txt                          # Qwen2 tokenizer merges
├── tokenizer_config.json               # Qwen2 tokenizer config
├── scripts/
│   └── onnx_inference_pure.py          # Inference script
└── prompts/
    ├── en_female_nova_greeting.wav     # Sample female voice
    └── en_male_onyx_greeting.wav       # Sample male voice

Model Files

ONNX Models (This Repository)

Tokenizer Files (Included)

Qwen2 tokenizer files are included in this repository:

• vocab.json - Vocabulary (3.54MB)
• merges.txt - BPE merges (1.54MB)
• tokenizer_config.json - Configuration

No additional downloads are required.

Architecture

1. Prompt Audio Processing
   ├── campplus.onnx → Speaker embedding (192-dim)
   ├── speech_tokenizer_v3.onnx → Speech tokens (for LLM context)
   └── librosa → Mel spectrogram (for Flow conditioning)

2. LLM Inference (Zero-Shot Mode)
   ├── text_embedding → [prompt_text + tts_text] embedding
   ├── llm_speech_embedding → Prompt speech token embedding
   ├── llm_backbone_initial → Initial pass (KV cache)
   ├── llm_backbone_decode → Decode steps (loop)
   └── llm_decoder → Logits → Token sampling

3. Flow Inference (Mel Generation)
   ├── flow_token_embedding → Token embedding
   ├── flow_pre_lookahead → Feature extraction
   ├── flow_speaker_projection → Speaker projection
   └── flow.decoder.estimator → DiT (10-step Euler)

4. HiFT Inference (Waveform)
   ├── hift_f0_predictor → F0 prediction
   ├── hift_source_generator → Source signal
   ├── STFT → Spectral decomposition
   ├── hift_decoder → Magnitude/phase prediction
   └── ISTFT → Waveform reconstruction

Quick Start

1. Create Environment

uv init cosyvoice-onnx --python 3.10
cd cosyvoice-onnx
uv add "onnxruntime==1.18.0" "numpy==1.26.4" "soundfile==0.12.1" "librosa==0.10.2" "transformers==4.51.3" "scipy==1.13.1" "huggingface_hub>=0.30.0"

Note: The --python 3.10 flag is required because onnxruntime 1.18.0 only supports Python 3.8-3.12.

2. Download Models

# Download everything (ONNX models + tokenizer + inference script + sample prompts)
uv run python -c "from huggingface_hub import snapshot_download; snapshot_download('ayousanz/cosy-voice3-onnx', local_dir='pretrained_models/Fun-CosyVoice3-0.5B/onnx')"

Note: All required files including the Qwen2 tokenizer are included in this repository. No additional downloads needed.

3. Run Inference

# English
uv run python pretrained_models/Fun-CosyVoice3-0.5B/onnx/scripts/onnx_inference_pure.py \
    --text "Hello, this is a test." \
    --prompt_wav pretrained_models/Fun-CosyVoice3-0.5B/onnx/prompts/en_female_nova_greeting.wav \
    --prompt_text "Hello, my name is Sarah. I'm excited to help you with your project today. Let me know if you have any questions." \
    --output output.wav

# Japanese
uv run python pretrained_models/Fun-CosyVoice3-0.5B/onnx/scripts/onnx_inference_pure.py \
    --text "こんにちは、今日はいい天気ですね。" \
    --prompt_wav pretrained_models/Fun-CosyVoice3-0.5B/onnx/prompts/en_female_nova_greeting.wav \
    --prompt_text "Hello, my name is Sarah. I'm excited to help you with your project today. Let me know if you have any questions." \
    --output output_ja.wav

Detailed Setup

Version Requirements

Important: The original CosyVoice (PyTorch version) also has the same version constraints. This is not specific to ONNX inference.

GPU Support (Optional)

uv remove onnxruntime && uv add "onnxruntime-gpu==1.18.0"

Note: Requires CUDA 11.8 or 12.x with cuDNN 8.x.

Command Line Arguments

Important: Zero-Shot Voice Cloning

CosyVoice3 uses zero-shot voice cloning. Both prompt audio AND prompt text are required:

• Prompt audio: Reference voice sample (3-10 seconds recommended)
• Prompt text: Transcript of what is spoken in the prompt audio

This provides better voice cloning quality than cross-lingual mode.

Performance

Note: CPU-only inference. GPU (CUDA) significantly improves performance.

Technical Notes

Pure ONNX Inference (PyTorch-Free)

This inference script runs completely without PyTorch. All processing is implemented using ONNX Runtime and NumPy/SciPy:

• Neural network inference: ONNX Runtime
• STFT/ISTFT: NumPy + SciPy (not PyTorch)
• Audio processing: librosa

HiFT Parameters (CosyVoice3 Specific)

Note: CosyVoice2 uses upsample_rates=[8, 8] (64x), but CosyVoice3 uses [8, 5, 3] (120x).

Expected STFT frames = mel_frames × 120 + 1

Precision Selection

• FP16: LLM and Flow components (memory efficient)
• FP32: HiFT components (numerical stability required), text embedding, speaker models

KV Cache

LLM uses split KV cache architecture:

• llm_backbone_initial: Generates initial KV cache from full context
• llm_backbone_decode: Updates KV cache with single token per step

Troubleshooting

ONNX Runtime version error with FP16 models

RuntimeException: Attempting to get index by a name which does not exist

Solution: Use onnxruntime==1.18.0. Newer versions (1.19+) have compatibility issues with FP16 models.

NumPy 2.x incompatibility

A module that was compiled using NumPy 1.x cannot be run in NumPy 2.x

Solution: Use numpy==1.26.4. This is a constraint shared with the original CosyVoice.

Tokenizer loading issues

If you encounter tokenizer loading errors, ensure you downloaded the complete repository including:

• vocab.json
• merges.txt
• tokenizer_config.json

Re-download with:

huggingface-cli download ayousanz/cosy-voice3-onnx --local-dir pretrained_models/Fun-CosyVoice3-0.5B/onnx

Language tags being pronounced

If you hear "<|en|>" or similar being spoken, remove the language tags from your text. CosyVoice3 automatically detects language.

License

Apache 2.0 (same as original CosyVoice)

Credits

• Original model: FunAudioLLM/Fun-CosyVoice3-0.5B-2512
• CosyVoice repository: FunAudioLLM/CosyVoice

Related Links

• CosyVoice GitHub
• ModelScope Model
• Paper: CosyVoice