OmniVoice-Urdu-3h

Next-Generation Fine-Tuned Model for High-Fidelity Urdu Zero-Shot Text-To-Speech, Voice Cloning & Voice Design

---

🎯 Model Overview

This checkpoint is a specialized Urdu adaptation of the foundational OmniVoice (k2-fsa/OmniVoice) framework. While the original architecture supports 600+ languages through flow-matching diffusion transformers, this fine-tuned release integrates ~3 hours of clean, diverse Urdu speech data to dramatically enhance phonetic accuracy, native prosody, and contextual phrasing.

---

🌟 Capabilities

🎤 1. Zero-Shot Voice Cloning

Instantly clone any target Urdu speaker using a 3–10 second clean reference clip — no additional fine-tuning or adaptation steps required.

audio = model.generate(
    text="آج موسم کتنا خوبصورت ہے,",
    ref_audio="speaker_ref.wav",
    # ref_text is automatically transcribed via Whisper if omitted
)

🎨 2. Voice Design

Synthesize natural Urdu speech with custom synthetic persona prompts — specify age, gender, pitch, and accent using plain text instructions.

audio = model.generate(
    text="یہ ایک تجربہ ہے آواز ڈیزائن کا,",
    instruct="female, young, high pitch",  # or "male, deep voice, pakistani accent"
)

🧠 3. Auto Voice Synthesis

Quickly synthesize natural, well-balanced Urdu speech without specifying any reference prompt or instructions.

audio = model.generate(text="میں آپ کی مدد کروں گا,")

🔧 Advanced Generation & Paralinguistic Controls

---

🚀 Quick Start

Installation

pip install omnivoice

# Optional GPU Acceleration via FlashInfer (Recommended for Production)
pip install flashinfer-python==0.6.15.post1 "flashinfer-jit-cache==0.6.15.post1+cu128" \
    --extra-index-url https://flashinfer.ai/whl/cu128/

Python API Usage

import soundfile as sf
import torch
from omnivoice import OmniVoice

# Load fine-tuned Urdu model
model = OmniVoice.from_pretrained(
    "mlwithabdullah/omnivoice-urdu-3h",
    device_map="cuda:0",  # Options: "cuda:0", "mps", "xpu", or "cpu"
    dtype=torch.float16,  # fp16 recommended for CUDA
    load_asr=True,  # Enables Whisper auto-transcription for reference audio
)

# 1. Zero-Shot Voice Cloning
audio = model.generate(
    text="پاکستان زندہ آباد، میرا وطن میرا جان,",
    ref_audio="urdu_speaker.wav",
)
sf.write("cloned_urdu.wav", audio[0], 24000)

# 2. Voice Design (Prompted Synthesis)
audio = model.generate(
    text="اس ماڈل نے بہت کام کیا ہے,",
    instruct="male, deep voice, pakistani accent",
)
sf.write("designed_urdu.wav", audio[0], 24000)

💻 Command Line Interface (CLI) Commands

# Voice Cloning via CLI
omnivoice-infer \
    --model mlwithabdullah/omnivoice-urdu-3h \
    --text "اردو میں بولنا اب آسان ہے," \
    --ref_audio urdu_ref.wav \
    --output cloned.wav

# Voice Design via CLI
omnivoice-infer \
    --model mlwithabdullah/omnivoice-urdu-3h \
    --text "آواز ڈیزائن کا تجربہ," \
    --instruct "female, soft, pakistani accent" \
    --output designed.wav

🌐 Local Interactive Web Interface

omnivoice-demo --share

---

📊 Fine-tuning Details

# Key training configuration settings:
init_from_checkpoint: "k2-fsa/OmniVoice"
steps: 5000
learning_rate: 5e-5

---

📈 Evaluation

A head-to-head benchmark evaluation was conducted comparing k2-fsa/OmniVoice (Base Model) against mlwithabdullah/omnivoice-urdu-3h (Fine-tuned) using a 5-sentence test suite spanning formal and casual Urdu speech registers.

💡 Key Takeaways & Insights

• Pronunciation & Intelligibility: Word Error Rate dropped dramatically from 22.3% down to 4.3%. While the base model occasionally struggled with corrupted scripts or Hindi script confusion on Urdu inputs, the fine-tuned checkpoint outputs consistent native Urdu speech.
• Speaker Similarity: A slight drop in cosine speaker similarity (0.80 → 0.76 SECS) is standard when adapting zero-shot models to target prosodic dynamics. 0.75+ SECS maintains strong voice-cloning fidelity.

📄 Detailed Sentence-Level Benchmark Breakdown

*Sentence 3 resulted in output corruption on the base model, leading to ASR script mismatch (Devanagari script return).

---

🔊 Audio Samples

Below is the direct audio playback comparison across all 5 benchmark sentences comparing the Base Model vs this Fine-Tuned Checkpoint.

---

💡 Best Practices for Urdu Inference

1. Reference Selection: Provide 3–10 seconds of clear, background-noise-free Urdu audio for best voice cloning accurate timbre.
2. Text Normalization: Pass normalize_text=True when generating speech containing digits, dates, or complex punctuation.
3. Hardware Acceleration: Install FlashInfer to achieve up to 2.6× speedups on modern NVIDIA GPUs.

---

📚 Citation & Acknowledgments

@article{zhu2026omnivoice,
  title={OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models},
  author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
  journal={arXiv preprint arXiv:2604.00688},
  year={2026}
}

Fine-tuned on Urdu speech data by Abdullah (mlwithabdullah/omnivoice-urdu-3h).

---

⚖️ License & Ethics

This model is licensed under Apache-2.0.

⚠️ Ethical Notice: Unconsented voice cloning, deepfake generation, or illegal impersonation is strictly prohibited. Users are fully responsible for ensuring compliance with applicable regional laws.

🤗 Base Model • 💻 GitHub Repository • 🎮 Interactive Space • 📄 Paper