This model is a fine-tuned version of microsoft/speecht5_tts on the VoxPopuli (nl) dataset. It achieves the following results on the evaluation set:
microsoft/speecht5_tts fine-tuned for Dutch text-to-speech (Unit 6 hands-on exercise of the Hugging Face Audio Course). The model predicts 80-bin log-mel spectrograms conditioned on a 512-d x-vector speaker embedding; a HiFi-GAN vocoder (microsoft/speecht5_hifigan) turns them into 16 kHz audio.
import torch
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor
repo = "Zorlu5454/speecht5_finetuned_voxpopuli_nl"
processor = SpeechT5Processor.from_pretrained(repo)
model = SpeechT5ForTextToSpeech.from_pretrained(repo)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")
# 512-d x-vector of the target speaker (speechbrain/spkrec-xvect-voxceleb, L2-normalised)
speaker_embeddings = torch.zeros(1, 512) # replace with a real x-vector
inputs = processor(text="hallo allemaal, ik praat nederlands.", return_tensors="pt")
speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=vocoder) # 16 kHz waveform
Dutch speech synthesis for experimentation. The SpeechT5 tokenizer is character-based and English: Dutch accented characters are mapped to their base letters (e.g. ë -> e) and numbers must be written out as words. Voices come from European Parliament speeches; do not use the model to imitate real people.
facebook/voxpopuli, nl train split (European Parliament
speeches). Only speakers with 100-400 utterances are kept (42 of 84
speakers), inputs with >= 200 tokens are dropped, and the rest is split 90/10:
7433 training and 826 evaluation utterances. Speaker embeddings are x-vectors from
speechbrain/spkrec-xvect-voxceleb.
The training loss is L1 on the mel spectrogram (before and after the post-net) + binary cross-entropy on the
stop token + guided attention loss on the cross-attention.
Code: https://github.com/emrezorlu1239/course-projects/tree/main/huggingface-audio-course (unit6_speecht5_tts/).
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 4.1503 | 4.2926 | 1000 | 0.4795 |
| 3.9650 | 8.5853 | 2000 | 0.4656 |
| 3.9370 | 12.8779 | 3000 | 0.4611 |
| 3.9210 | 17.1678 | 4000 | 0.4596 |
model.safetensors 578 MBtraining_args.bin 5.5 kBmicrosoft/speecht5_tts