SpeechT5 fine-tuned on Polish VoxPopuli

A learning experiment, not a production TTS model. It answers one question: what happens when you fine-tune an English text-to-speech model on a small amount of noisy Polish speech?

Made while working through Unit 6 of the Hugging Face Audio Course as part of a longer audio ML learning plan. Each published change is a separate tagged version, so the effect of each change can be heard and compared.

Listen

Text: "Cześć, to jest test wygenerowanego głosu w języku polskim. Możesz powiedzieć, że to jest bardzo fajne, prawda?" Same text, same speaker embedding, base model vs each fine-tuned version:

The base model, v1 and v2 read the text with Polish letters mapped to ASCII; v3 reads the real letters. The exact input of each model is in text_to_synthesize.txt.

How it works

SpeechT5 is an encoder-decoder Transformer. Text goes in as characters, the decoder predicts an 80-bin log-mel spectrogram two frames per step, and a separate HiFi-GAN vocoder turns the spectrogram into a 16 kHz waveform. Whose voice it speaks in is set by a 512-dimensional x-vector from a speaker verification model, passed in at inference.

text -> characters -> SpeechT5 encoder -> decoder (+ x-vector) -> log-mel -> HiFi-GAN -> 16 kHz audio

This is the oldest family of voice cloning (speaker embedding conditioning, as in SV2TTS, 2018). The x-vector keeps the timbre of a voice and drops its prosody, so outputs sound like "someone similar" rather than the reference speaker.

Versions

Load a version with both the model and the processor from the same tag. From v3 on, text must be encoded per character as below, not with processor(text=...): with added tokens the SentencePiece tokenizer inserts a word boundary after every Polish letter and drops the one before it.

import torch
from datasets import load_dataset
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor

repo, version = "wozniakos10/speecht5_finetuned_voxpopuli_pl", "v3-polish-tokens"
processor = SpeechT5Processor.from_pretrained(repo, revision=version)
model = SpeechT5ForTextToSpeech.from_pretrained(repo, revision=version)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

# any 512-d x-vector works; this one is from the CMU Arctic set used in the course
xvectors = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation", revision="refs/convert/parquet")
speaker_embeddings = torch.tensor(xvectors[7306]["xvector"]).unsqueeze(0)

vocab = processor.tokenizer.get_vocab()


def encode_text(text: str) -> list[int]:
    pieces = ["▁"] + ["▁" if ch == " " else ch for ch in text.lower()]
    return [vocab[piece] for piece in pieces] + [processor.tokenizer.eos_token_id]


input_ids = torch.tensor([encode_text("Cześć, to jest test polskiego głosu.")])
speech = model.generate_speech(input_ids, speaker_embeddings, vocoder=vocoder)  # 16 kHz

# v1 only knows ASCII: map ą->a, ć->c, ę->e, ł->l, ń->n, ó->o, ś->s, ź->z, ż->z before encode_text

Training data

VoxPopuli, Polish train split: European Parliament speeches. It is an ASR corpus, not a TTS one: room reverb, applause, spontaneous speech.

• Rows with digits dropped (2 of 34,665 transcripts).
• Speakers kept only with 100 to 400 clips, so no single voice dominates and no voice is too rare to learn.
• Up to 10,000 clips sampled, then clips over 200 input tokens dropped; 90/10 train/eval split.
• Speaker embeddings: speechbrain/spkrec-xvect-voxceleb, L2-normalised.

Training

Loss is SpeechT5's own: L1 on the mel spectrogram plus stop-token BCE. The best eval loss checkpoint is kept.

v1: the course recipe

lr 1e-5 linear with 10% warmup, 4 per device x 8 accumulation = 32, 10 epochs, fp16, gradient checkpointing.

Eval loss was still falling at the end: v1 is undertrained. Eval loss below train loss is expected, dropout is on during training and train loss is averaged over the whole epoch.

v2: same model and text handling, a training recipe for a 32 GB GPU

The v2 weights were overwritten on disk before they were uploaded, so v2 has no tag and cannot be loaded. Its samples are kept at commit 05c5c30 and its numbers are below.

Same data as v1. lr 5e-5 with a cosine schedule, batch 32 in one step (no accumulation, no gradient checkpointing), evaluation every 250 steps with early stopping.

Eval loss drops from 0.455 to 0.427 and is flat from about step 3500 on, so v2 has converged for this data and this text handling. The loss gap looks small, but the audio difference is large: loss is an average over every mel frame and says little about what the ear notices. More training will not fix the remaining errors, which come from the ASCII mapping below.

v3: Polish letters as tokens

Same data and recipe as v2. The nine letters ą ć ę ł ń ó ś ź ż get their own tokens (vocabulary 81 to 90). Each new embedding row starts as a copy of the nearest letter (ż and ź from z, ó from u because that is how it is pronounced), so training starts from v2's behaviour and only has to learn the difference. Text is encoded per character, bypassing the tokenizer.

Findings: what the Polish tokens changed

• Loss: nothing. 0.4266 vs 0.4269 for v2. Diacritic sounds are a small share of all mel frames.

• The test sentence: no audible difference. The encoder sees whole words, and after stripping diacritics Polish words rarely collide: moze only ever meant może in the v2 training data, so v2 had already learned to say it with ż from context. The ASCII mapping only loses information on words that become identical without diacritics.

• Minimal pairs: ł works. Generated with v3 from Polish and from ASCII input, transcribed by a CTC model without a language model (jonatasgrosman/wav2vec2-large-xlsr-53-polish):

  Input	Heard
  łaska i laska (Polish tokens)	łaska i laska
  laska i laska (ASCII)	lasko i lasko
  to jest żona, a to jest zona (Polish tokens)	to jest żona to jest żona
  to jest zona, a to jest zona (ASCII)	to jes żona to jest żona

  ł is distinguished. ż/z is inconclusive: both inputs come out as żona. Pairs of two or three words (sąd i sad, kość i kosc) produced unintelligible audio for both inputs: the model was trained on long parliament segments and breaks on very short input. Whisper was not used here because its decoder acts as a language model and corrects rare words towards common ones.

Limitations

• Polish diacritics are lost in v1 and v2. ż and ź both become z, ł becomes l, ą and ę lose nasality, and ó is read as o instead of u, because the base tokenizer has none of these letters. v3 has them as tokens; only ł is confirmed to be pronounced differently so far.
• Very short input breaks. Two or three words often come out unintelligible. Use full sentences.
• No numbers. Write them out as words.
• Noisy voice. Parliament acoustics are in the training data and come out in the audio.
• Weak speaker identity. The x-vector model was trained on English VoxCeleb; it separates Polish voices less well, and an embedding carries timbre, not speaking style.
• Loss is not quality. L1 on mel rewards blurry, averaged spectrograms. Versions should be compared by listening and by the character error rate of an ASR pass over the output, not by loss alone.

Intended use

Learning and comparison of fine-tuning choices. Not for production, and not for imitating any real person. VoxPopuli speakers are real Members of the European Parliament; do not use this model to make anyone appear to say something they did not.

Credits

• Base model: SpeechT5 (Ao et al., 2022), microsoft/speecht5_tts, MIT.
• Data: VoxPopuli (Wang et al., 2021).
• Recipe: Hugging Face Audio Course, Unit 6.