CosyVoice-300M ONNX

Source

Precision and Packaging

Export tooling, precision, and quantization are recorded in the Source table above. This packaging mirror does not publish independent parity benchmarks; validate on your target execution provider before production use.

Intended Uses

• Cross-platform TTS deployment without a PyTorch dependency
• Edge/embedded inference via ONNX Runtime on CPU, GPU, or NPU
• Lower-latency serving with execution-provider acceleration (validate before relying on it)
• Research on TTS model optimization and format conversion

Safety and Responsible Use

CosyVoice-300M is a zero-shot voice-cloning model capable of producing realistic speech that closely matches a target speaker from a short reference clip.

• Do not use to impersonate real individuals without their explicit consent.
• Do not generate synthetic speech intended to deceive listeners about a speaker's identity.
• Disclose AI-generated audio where listeners would reasonably expect a human voice.
• Users are responsible for compliance with applicable laws governing synthetic media and voice cloning in their jurisdiction.

Limitations & Biases

1. Autoregressive decoding: The token generator is exported in teacher-forced mode; step-by-step decoding with KV-cache requires custom ONNX graphs or runtime loops
2. STFT approximation: The vocoder uses DFT matrix multiplication instead of FFT, introducing minor numerical differences (max 1.98 absolute)
3. Frontend models: campplus.onnx (speaker encoder) and speech_tokenizer_v1.onnx (speech tokenizer) are from the original repo, not part of this conversion
4. No training data: This is a format conversion, not a trained model — no new training was performed

What's in this repo

ONNX Models (ready for inference)

Pre-existing ONNX (from original CosyVoice repo)

PyTorch weights

Config

Usage

import numpy as np
import onnxruntime as ort

# Load models
text_encoder = ort.InferenceSession("llm/text_encoder.onnx")
token_generator = ort.InferenceSession("llm/token_generator.onnx")
flow_encoder = ort.InferenceSession("flow/encoder.onnx")
hift_source = ort.InferenceSession("hift/source.onnx")
hift_vocoder = ort.InferenceSession("hift/vocoder.onnx")

# Example: encode text tokens
text_tokens = np.array([[101, 2023, 3042, 102]], dtype=np.int64)  # tokenized input
text_lengths = np.array([4], dtype=np.int64)
encoded = text_encoder.run(None, {"text_tokens": text_tokens, "text_lengths": text_lengths})[0]
# encoded shape: (1, 4, 1024)

Benchmarks

[!NOTE] All figures below were measured by the repository maintainer on a single NVIDIA RTX 5070 machine (PyTorch 2.12, onnxruntime 1.26). They are illustrative, not guaranteed, and were not independently reproduced. Re-measure on your own hardware and execution provider before relying on them. Numbers describe latency/throughput, not output quality.

Inference Latency (NVIDIA RTX 5070)

All latencies report mean (p50, p95) in milliseconds.

Notes (maintainer measurements, single machine):

• Max observed GPU speedup vs CPU: 59.5x (Text Encoder, seq=500)
• Max observed TRT RTX speedup over CUDA: 14.3x (Flow Encoder, T=50)
• CUDA outperformed TRT RTX on the vocoder (STFT ops) in this run
• All components achieved RTF < 1.0 on both CPU and GPU in this run

PyTorch vs ONNX Runtime (GPU)

Numerical verification (maintainer-run)

ONNX exports were checked against PyTorch reference outputs by the maintainer:

Model Sizes

Model Architecture

CosyVoice-300M uses a three-stage pipeline:

1. LLM (TransformerLM): ConformerEncoder text encoder (6 blocks, in=512, out=1024) + TransformerEncoder language model (14 blocks, dim=1024) + linear decoder (-> 4097 speech tokens)
2. Flow (MaskedDiffWithXvec): ConformerEncoder (6 blocks, out=512) + InterpolateRegulator (length expansion) + ConditionalCFM decoder with ConditionalDecoder estimator (channels=[256,256], 4 blocks, 12 mid_blocks)
3. HiFT (HiFTGenerator): SourceModuleHnNSF (harmonic + noise excitation) + ConvRNNF0Predictor + convolutional decoder with upsampling (2 stages, 8x each) + ISTFT

Notes

• All ONNX models were exported with CPU-only PyTorch 2.12 and verified with onnxruntime 1.26.
• The vocoder uses a DFT-matrix-based ISTFT (instead of FFT) for ONNX compatibility. Results are numerically close but not bit-identical to the PyTorch version.
• The hift/vocoder.onnx file has an accompanying hift/vocoder.onnx.data external data file (~51 MB).

Citation

@article{du2024cosyvoice,
  title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},
  author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and others},
  journal={arXiv preprint arXiv:2407.05407},
  year={2024}
}

Disclaimer

This is an unofficial ONNX conversion. The original model and architecture are by Alibaba's FunAudioLLM team. For the official implementation, see github.com/FunAudioLLM/CosyVoice.

Additional FP16 candidate bundles

Two optional, experimental bundles are included alongside the existing ONNX files:

• trt_rtx_fp16/: mixed-precision FP16 candidate produced with NVIDIA ModelOpt.
• trt_rtx_fp16_graphsurgeon/: graph-cleaned copy of that candidate using ONNX GraphSurgeon.

The root FP32 and quantized artifacts remain unchanged. See verification_report.json for checks and limits. These candidates were exercised with ONNX Runtime's CUDA Execution Provider; the TensorRT-RTX Execution Provider itself was not available for validation, so TensorRT-RTX compatibility is not claimed. Keep every .onnx.data sidecar next to its graph.