Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine

One voice. Any language.

                                       

Confucius4-TTS is an advanced LLM-based text-to-speech (TTS) system designed for multilingual and cross-lingual speech synthesis. Built on a speech encoder + large language model (LLM) architecture, Confucius4-TTS enables high-quality speech generation while preserving speaker identity across languages. You can try our online demo at https://confucius4-tts.youdao.com/gradio.

✨ Key Features

• 14 Languages Supported: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay and Vietnamese (more coming soon)
• Unconstrained Voice Cloning: No reference transcript required
• Cross-Lingual Voice Transfer: Unaccented speech synthesis across 14 languages
• Zero-Shot Voice Transfer: Clone voices without additional training
• Seamless Emotion Transfer: Clone the feeling, not just the voice
• Robust Generalization: Stable performance in real-world multilingual scenarios

With strong cross-lingual generalization, Confucius4-TTS allows users to seamlessly switch languages while keeping the same voice, delivering fluent, natural, and expressive speech.

Contents

• Installation
• Inference
• Training
• Performance
• Citation

🛠 Installation

Requirements

• Python 3.10
• CUDA 12.6

Setup

1. Clone the repository:

git clone https://github.com/netease-youdao/Confucius4-TTS.git
cd Confucius4-TTS

2. Create and activate a conda environment:

conda create -n confuciustts python=3.10 -y
conda activate confuciustts

3. Install dependencies:

pip install -r requirements.txt

🚀 Inference

Use the provided example.py script for zero-shot TTS synthesis:

python example.py \
    --prompt_wav path/to/reference.wav \
    --text "Your text to synthesize" \
    --lang en \
    --out output.wav \
    --config config/inference_config.yaml

You can also use the Python API directly:

import torch
import torchaudio
from confuciustts.cli.inference import ConfuciusTTS

model = ConfuciusTTS(
    config_path="config/inference_config.yaml",
    device="cuda" if torch.cuda.is_available() else "cpu",
)

audio = model.generate(
    text="Hello, welcome to Confucius4-TTS.",
    lang="en",
    prompt_wav="path/to/reference.wav",
    verbose=True,
)

torchaudio.save("output.wav", audio.cpu(), model.sample_rate)

🚀 Fine-Tuning

Confucius4-TTS follows a "speech encoder + LLM" architecture. The training pipeline covers two modules:

• Text2Semantic (T2S): generates semantic token sequences from text and speaker conditioning.
• Semantic2Acoustic (S2A): a flow-matching model that converts semantic tokens into mel spectrograms.

1. Prepare Pretrained Models

Download the two external models:

# Wav2Vec2-BERT (speaker conditioning & semantic feature extraction)
huggingface-cli download facebook/w2v-bert-2.0 \
    --local-dir pretrained/w2v-bert-2.0

# Amphion MaskGCT (semantic codec implementation)
git clone https://github.com/open-mmlab/Amphion.git external/Amphion

After downloading, your directory should look like:

checkpoints/
├── t2s_model.safetensors        # pretrained T2S weights
├── s2a_model.pt                 # pretrained S2A weights
├── wav2vec2bert_stats.pt        # semantic feature normalization statistics
├── special_tokens_map.json      # tokenizer files
├── tokenizer.json
├── tokenizer.model
└── tokenizer_config.json
pretrained/
├── w2v-bert-2.0/                # Wav2Vec2-BERT model
└── campplus/
    └── campplus_cn_common.bin   # CAMPPlus speaker encoder checkpoint
external/
└── Amphion/                     # MaskGCT semantic codec implementation

2. Prepare Training Data

Training data is provided as TSV files (tab-separated, no header) with the following 5 columns:

Configure the train/validation paths in config/train_t2s.yaml:

data:
  train_data_path:
    - data/train.tsv
  val_data_path:
    - data/val.tsv

3. Launch T2S Training

Set the pretrained T2S checkpoint path in config/train_t2s.yaml:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors

Single-node training:

python -m confuciustts.cli.train_t2s -c config/train_t2s.yaml

4. Launch S2A Training

Set the checkpoint paths in config/train_s2a.yaml. t2s_checkpoint points to the frozen T2S backbone; s2a_checkpoint is optional and can be used to resume from a pretrained S2A model:

paths:
  t2s_checkpoint: checkpoints/t2s_model.safetensors
  s2a_checkpoint: checkpoints/s2a_model.pt   # optional: resume from pretrained S2A

Single-node training:

python -m confuciustts.cli.train_s2a -c config/train_s2a.yaml

During S2A training, the T2S model, speaker encoder (Wav2Vec2-BERT), and style encoder (CAMPPlus) are all frozen. Only the flow-matching S2A model is trained.

📊 Performance

Confucius4-TTS achieves competitive results on multilingual and cross-lingual zero-shot TTS benchmarks, with strong intelligibility and speaker similarity across multiple languages.

Lower is better for WER/CER (↓), and higher is better for SIM (↑).

CV3-eval Cross-lingual

CV3-eval Cross-lingual Results (click to expand)

† Requires reference text.

X-Voice Benchmark

X-Voice Cross-lingual Results (click to expand)

† Requires reference text.

Seed-TTS-eval

Seed-TTS-eval English & Chinese Zero-shot Results (click to expand)

† Requires reference text.

MiniMax-MLS-Test

MiniMax-MLS-Test Results (click to expand)

† Requires reference text.

---

Acknowledgements

Confucius4-TTS builds on the following open-source projects:

• Qwen3-TTS — Speaker encoder (ECAPA-TDNN) and text embedding projector architectures
• CosyVoice — Text normalization pipeline
• Amphion / MaskGCT — Semantic codec implementation
• w2v-BERT 2.0 — Semantic feature extraction and speaker conditioning
• Seed-VC — Flow matching architecture reference
• BigVGAN — High-fidelity neural vocoder for mel-spectrogram to waveform synthesis

---

Citation

If you find Confucius4-TTS useful in your research or project, please consider citing:

@misc{wang2026confucius4tts,
  title         = {Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder},
  author        = {Huaxuan Wang and Huimin Wang and Ruiyu Zhang and Yingjie Li and Yitao Duan},
  year          = {2026},
  eprint        = {2608.11650},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2608.11650}
}