SpeechT5 — LJSpeech course practice

Actual fine-tuning of microsoft/speecht5_tts for the Audio Course Unit 6 practice assignment.

Data and method

176 public LJSpeech examples between one and nine seconds long, text under 180 characters. Fixed train/test split: 160/16, seed 42. Audio resampled to 16 kHz; speaker embeddings extracted with speechbrain/spkrec-xvect-voxceleb and normalized. Training used 80 optimizer steps, batch size 2 and gradient accumulation 2 (two passes over the training subset).

Results and limitations

Held-out spectrogram loss: 0.5500234365463257. Training loss: 0.6324256479740142. No MOS or human listening assessment has been performed; this loss is not a measure of perceptual speech quality. This is a small educational fine-tuning run, not a production model or a new-language model. The model remains an English TTS system. The course provider has not yet assessed an uploaded version.

The saved speaker_embedding.npy is derived from a public dataset example, not from the account holder's voice. To synthesize, load the checkpoint with SpeechT5ForTextToSpeech, load its processor, and use microsoft/speecht5_hifigan as the vocoder.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• training_steps: 80

Training results

Framework versions

• Transformers 4.57.6
• Pytorch 2.14.1
• Datasets 3.6.0
• Tokenizers 0.22.2