Loading…
Loading…
This repository hosts a high-fidelity fine-tuned version of the Chatterbox TTS model, specifically optimized for the Finnish language. By leveraging a multilingual base and large-scale Finnish data, we achieved exceptional zero-shot generalization to unseen speakers, surpassing commercial-grade quality thresholds.
The following metrics were calculated on Out-of-Distribution (OOD) speakers who were strictly excluded from the training and validation sets. This measures how well the model can speak Finnish in voices it has never heard before.
| Metric | Baseline (Original Multilingual) | Fine-Tuned (Step 986) | Improvement |
|---|---|---|---|
| Avg Word Error Rate (WER) | 28.94% | 2.76% | ~10.5x Accuracy Increase |
| Mean Opinion Score (MOS) | 2.29 / 5.0 | 4.34 / 5.0 | +2.05 Quality Points |
Note: MOS was evaluated using the Gemini 3 Flash API, and WER was calculated using Faster-Whisper Finnish Large v3. The 4.34 MOS indicates a "Professional Grade" output comparable to human speech.
Listen to the difference between the generic multilingual baseline and our high-fidelity Finnish fine-tuning. These samples are from speakers never seen during training.
| Speaker ID | Baseline (Generic Multilingual) | Fine-Tuned (Finnish Golden) |
|---|---|---|
| cv-15_11 | ||
| cv-15_16 | ||
| cv-15_2 |
The model was trained on a diverse corpus of 16,604 samples to capture the nuances of Finnish phonetics, including vowel length and gemination.
cv-15_11, cv-15_16, cv-15_2) were strictly excluded from training to ensure valid OOD testing.attribution.csv.As a separate research phase, we tested the model's capacity for deep voice cloning by fine-tuning the Phase 1 base on a specific high-quality Finnish dataset (GrowthMindset).
We used sweep_params.py to identify the "Golden Settings" for the most natural Finnish inference. By evaluating against holdout samples and everyday phrases, we achieved a peak quality of 4.63 MOS.
Best Parameters for Finnish:
repetition_penalty: 1.5 (Balanced for Finnish long vowels)temperature: 0.8exaggeration: 0.5cfg_weight: 0.3Note: The single-speaker weights are not included in this repository.
AlignmentStreamAnalyzer to support Finnish phonology.Open this repo in VS Code with the Dev Containers extension. Everything — dependencies, base model weights, GPU detection — is handled automatically by postCreateCommand.
# 1. Clone (with LFS for model weights)
git clone https://huggingface.co/Finnish-NLP/Chatterbox-Finnish
cd Chatterbox-Finnish
# 2. Install dependencies (auto-detects your GPU architecture)
bash install_dependencies.sh
# 3. Download pretrained base models from ResembleAI
python setup.py
# 4. Run inference
python inference_example.py
GPU compatibility: The install script detects your GPU and picks the right PyTorch build automatically:
- Blackwell (sm_120+) e.g. RTX PRO 6000 → PyTorch 2.10.0 + CUDA 12.8
- Older GPUs (A100, RTX 30/40xx, etc.) → PyTorch 2.5.1 + CUDA 12.4
import torch
import soundfile as sf
from src.chatterbox_.tts import ChatterboxTTS
from safetensors.torch import load_file
device = "cuda" if torch.cuda.is_available() else "cpu"
# 1. Load the base engine
engine = ChatterboxTTS.from_local("./pretrained_models", device=device)
# 2. Inject Finnish fine-tuned weights
checkpoint = load_file("./models/best_finnish_multilingual_cp986.safetensors")
t3_state = {k[3:] if k.startswith("t3.") else k: v for k, v in checkpoint.items()}
engine.t3.load_state_dict(t3_state, strict=False)
# 3. Generate with Finnish-optimized parameters
wav = engine.generate(
text="Tervetuloa kokeilemaan hienoviritettyä suomenkielistä Chatterbox-puhesynteesiä.",
audio_prompt_path="./samples/reference_finnish.wav",
repetition_penalty=1.2,
temperature=0.8,
exaggeration=0.6,
)
sf.write("output.wav", wav.squeeze().cpu().numpy(), engine.sr)
Or just run the included example script directly:
python inference_example.py # outputs output_finnish.wav
ResembleAI/chatterbox