🦜 Jerry’s Voice Clone — SpeechT5 Fine-Tuned Model

This model fine-tunes Microsoft’s SpeechT5 to reproduce Jerry Cuomo’s speaking voice.
It was trained entirely in Google Colab using open-source tools from 🤗 Transformers, SpeechBrain, and Librosa.

🧩 Model Description

A PyTorch-only text-to-speech model that generates speech from text using a fine-tuned version of SpeechT5 and a HiFi-GAN vocoder. The speaker embedding was extracted with SpeechBrain’s spkrec-xvect-voxceleb model.

🎯 Intended Uses & Limitations

• Designed for educational and experimental use in AI voice cloning.
• Not intended for impersonation or commercial deployment.
• Works best with 16 kHz input and the provided HiFi-GAN vocoder.

🧠 Training and Evaluation Data

• Base model: microsoft/speecht5_tts
• Vocoder: microsoft/speecht5_hifigan
• Dataset: Custom 16 kHz WAV clips + transcriptions.csv
• Steps: ≈ 2 000
• Batch: 2 × 8 (accumulated)
• Learning rate: 1e-4
• Optimizer: AdamW (default Hugging Face Trainer)

⚙️ Training Procedure

Key hyperparameters

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: AdamW (betas = 0.9/0.999, eps = 1e-08)
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 200
• training_steps: 2000
• mixed_precision_training: Native AMP

📦 Framework Versions

• Transformers 4.57.0
• PyTorch 2.8.0 + cu126
• Datasets 2.20.0
• Tokenizers 0.22.1

🎙 Example Usage

from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan
import torch, soundfile as sf

repo = "YOUR_USER/speecht5-jerry"
processor = SpeechT5Processor.from_pretrained(repo)
model = SpeechT5ForTextToSpeech.from_pretrained(repo)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

text = "Hello, this is Jerry from the Wild Ducks podcast."
inputs = processor(text=text, return_tensors="pt")

# Example speaker embedding (load or extract from dataset)
import numpy as np
speaker = torch.tensor(np.load("speaker_emb.npy")).unsqueeze(0)

speech = model.generate_speech(
    inputs["input_ids"], speaker_embeddings=speaker, vocoder=vocoder
)
sf.write("jerry_demo.wav", speech.numpy(), 16000)