Naija CSM Fine-tuned Text-to-Speech Model

A fine-tuned CSM (Conditional Speech Model) for generating natural-sounding Nigerian English speech from text. This model specializes in producing TTS output with authentic Nigerian accent and pronunciation patterns.

Model Details

Model Description

This is a fine-tuned Text-to-Speech (TTS) model based on the CSM (Conditional Speech Model) architecture, specifically trained on Nigerian English audio data to generate natural-sounding speech with Nigerian accent and pronunciation patterns. The model uses LoRA (Low-Rank Adaptation) fine-tuning for efficient training and maintains the multi-speaker capabilities of the base model.

• Developed by: Sam4rano
• Model type: Text-to-Speech (TTS) / Conditional Speech Model
• Language(s) (NLP): English (Nigerian variant)
• License: Apache 2.0
• Finetuned from model: unsloth/csm-1b

Model Sources

• Repository: https://huggingface.co/Sam4rano/naija-csm-finetuned-model
• Base Model: unsloth/csm-1b
• Training Framework: Unsloth

Uses

Direct Use

This model can be used to generate Nigerian-accented English speech from text input. It's particularly useful for:

• Creating TTS systems for Nigerian applications
• Generating natural-sounding Nigerian English audio for educational content
• Voice synthesis for Nigerian-focused applications and services

Downstream Use

• Audiobook narration with Nigerian accent
• Educational tools for Nigerian English pronunciation

Out-of-Scope Use

• This model is not suitable for generating speech in languages other than Nigerian English
• Should not be used for creating misleading, deceptive, or harmful audio content
• Not recommended for real-time applications without proper optimization
• Not suitable for other English variants or accents without additional fine-tuning

Bias, Risks, and Limitations

• Accent Specificity: Trained primarily on Nigerian English, may not perform well for other English variants
• Speaker Diversity: Quality may vary depending on the diversity of speakers in the training data
• Audio Quality: Output quality depends on the quality and diversity of the training dataset
• Computational Requirements: Requires GPU for optimal inference speed
• Limited Training: Model was trained for only 60 steps as a demonstration - production use may require additional training

Recommendations

Users should be aware that this model is optimized for Nigerian English and may not generalize well to other accents or languages. Consider additional training with more data and epochs for improved performance.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import CsmForConditionalGeneration, AutoProcessor
import torch
import soundfile as sf

# Load the model and processor
model = CsmForConditionalGeneration.from_pretrained("Sam4rano/naija-csm-finetuned-model")
processor = AutoProcessor.from_pretrained("Sam4rano/naija-csm-finetuned-model")

# Generate speech
text = "Welcome to Nigeria, the giant of Africa!"
speaker_id = 0

# Method 1: Simple text input
inputs = processor(f"[{speaker_id}]{text}", add_special_tokens=True)
audio_values = model.generate(
    **inputs,
    max_new_tokens=125,  # 125 tokens ≈ 10 seconds of audio
    output_audio=True
)

# Convert to audio array and save
audio = audio_values[0].to(torch.float32).cpu().numpy()
sf.write("nigerian_speech.wav", audio, 24000)

# Method 2: Conversation format (recommended)
conversation = [
    {"role": str(speaker_id), "content": [{"type": "text", "text": text}]},
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
)

audio_values = model.generate(
    **inputs,
    max_new_tokens=125,
    output_audio=True
)

audio = audio_values[0].to(torch.float32).cpu().numpy()
sf.write("nigerian_speech.wav", audio, 24000)

# For voice consistency, provide audio context
sample_audio = your_reference_audio_array  # numpy array at 24kHz
sample_text = "Your reference text"

conversation_with_context = [
    {"role": str(speaker_id), "content": [
        {"type": "text", "text": sample_text},
        {"type": "audio", "path": sample_audio}
    ]},
    {"role": str(speaker_id), "content": [{"type": "text", "text": text}]},
]

inputs = processor.apply_chat_template(conversation_with_context, tokenize=True, return_dict=True)
audio_values = model.generate(**inputs, max_new_tokens=125, output_audio=True)

Training Details

Training Data

The model was trained on a custom Nigerian English audio dataset containing text-audio pairs with speaker information. The dataset was preprocessed to ensure consistent formatting and quality.

Data characteristics:

• Language: Nigerian English
• Audio format: 24kHz sampling rate
• Speaker handling: Multi-speaker support with consistent speaker IDs
• Text processing: Tokenized with special speaker tokens

Training Procedure

Preprocessing

The training data underwent several preprocessing steps:

• Audio files resampled to 24kHz sampling rate
• Maximum audio length: 240,001 samples (approximately 10 seconds)
• Maximum text length: 256 tokens
• Speaker IDs converted to consistent string format
• Audio padding to maximum length for batch processing
• Text padding with attention masks

Training Hyperparameters

• Training regime: Mixed precision (fp16/bf16) with LoRA fine-tuning
• Base model: unsloth/csm-1b
• LoRA Configuration:
  • LoRA rank (r): 32
  • LoRA alpha: 32
  • LoRA dropout: 0
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4
• Batch size: 2 (per device)
• Gradient accumulation steps: 4
• Training steps: 60
• Warmup steps: 5
• Optimizer: AdamW 8-bit
• Weight decay: 0.01
• LR scheduler: Linear
• Random seed: 3407

Speeds, Sizes, Times

• Training time: Approximately 2-5 minutes on Tesla T4 GPU
• Memory usage: ~8-12GB GPU memory during training
• Model size: LoRA adapters only (~few MB), requires base model for full functionality
• Hardware: Google Colab Tesla T4 GPU (16GB VRAM)

Evaluation

Testing Data, Factors & Metrics

Testing Data

Due to the demonstration nature of this training (60 steps), formal evaluation was limited to basic functionality testing and audio generation quality assessment.

Factors

• Audio quality and naturalness
• Speaker consistency
• Pronunciation accuracy for Nigerian English
• Text-to-speech alignment

Metrics

• Qualitative audio assessment
• Model loading and inference functionality
• Generation capability verification

Results

The model successfully generates Nigerian-accented English speech and maintains the multi-speaker capabilities of the base model. However, extended training would be beneficial for production use.

Summary

This model serves as a proof-of-concept for Nigerian English TTS and demonstrates the effectiveness of LoRA fine-tuning for speech synthesis models.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA Tesla T4 GPU
• Hours used: ~0.1 hours (2-5 minutes)
• Cloud Provider: Google Colab
• Compute Region: Variable
• Carbon Emitted: Minimal due to short training time

Technical Specifications

Model Architecture and Objective

• Architecture: CSM (Conditional Speech Model) - transformer-based text-to-speech model
• Objective: Conditional generation of speech audio from text with speaker conditioning
• Input: Text tokens + speaker ID
• Output: Audio waveform at 24kHz sampling rate
• Fine-tuning method: LoRA (Low-Rank Adaptation)

Compute Infrastructure

Training was conducted using the Unsloth framework for memory-efficient fine-tuning.

Hardware

• Google Colab Tesla T4 GPU (16GB VRAM)
• CUDA-compatible environment
• Memory-optimized training with Unsloth

Software

• Python 3.10+
• PyTorch 2.0+
• Transformers 4.52.3
• Unsloth framework
• Additional dependencies: soundfile, datasets, accelerate

Citation

BibTeX:

Glossary

• CSM: Conditional Speech Model - A transformer-based architecture for text-to-speech synthesis
• LoRA: Low-Rank Adaptation - An efficient fine-tuning method that updates only a small subset of model parameters
• TTS: Text-to-Speech - Technology that converts written text into spoken audio
• Nigerian English: The variety of English spoken in Nigeria, with distinct pronunciation and linguistic features

Model Card Authors

Sam4rano

Model Card Contact

For questions or issues regarding this model, please create an issue in the model repository or contact through Hugging Face.