voicetut-egyptian-tashkeel

VoiceTut-TTS (OmniVoice 0.6B, Egyptian Arabic) fine-tuned on Rabe3/egyptian-arabic-tts-diacritized: 17,546 Egyptian Arabic clips (58.6 hours, 9 speakers) whose text carries short vowels (tashkeel) on about half of the words. Write the vowels in the text and the model reads them. The voice prompt was kept in training, so VoiceTut's 19 built-in voices and cloning from a reference clip still work.

Listen

The sentences are the ones on the card of the emotion model. The vowels in the table were written by hand (every word). Plain text without vowels works as before.

Morning with coffee (voice: Ahmed)

Shopping with friends (voice: Mahmoud)

Joking with friends (voice: Tareq)

Remembering someone who passed (voice: Mariam)

Usage

Install OmniVoice from github.com/k2-fsa/OmniVoice, then:

import json, torch, soundfile as sf
from huggingface_hub import snapshot_download
from omnivoice import OmniVoice

refs = snapshot_download("mohammedaly22/VoiceTut-TTS", allow_patterns=["reference_speakers/*"])
voice = json.load(open(f"{refs}/reference_speakers/references.json", encoding="utf-8"))[0]    # any of the 19 built-in voices, or your own clip and its transcript
model = OmniVoice.from_pretrained("masterofaudio2077/voicetut-egyptian-tashkeel", device_map="cuda:0", dtype=torch.float16)
text = "صَبَاحْ الْخِيرْ، انَا لِسَّهْ مِشْ قَادِرْ افْتَحْ عَيْنِي."      # vowels on some or all words
audio = model.generate(text=text, language="arz", ref_audio=f"{refs}/{voice['audio_path']}", ref_text=voice["reference_text"])[0]
sf.write("out.wav", audio, 24000)

Vowels can be added to any text with a diacritizer. The training text had vowels on about half of the words (median 0.52), so partly voweled text is the case this model was trained for.

Results

24 held-out sentences read with the 19 built-in voices in turn, plus 20 pairs of words with the same consonants and different vowels (two voices), scored on the output audio.

The "heard" vowels come from Cohere-Speech-Tashkeel-2B, which has its own error (6.6% DER on its test set), so the rates are relative, not absolute.

Training

• Full fine-tune of VoiceTut-TTS, 13500 steps on 2x T4, Lion optimizer (lr 3e-6, weight decay 0.03), fp16. This is step 10000.

• 17,546 clips, 58.6 h, text column text_merged; 396 clips held out (their sentences removed from training). Hours per speaker:

• lahgtna_eg_spk00000: 6.6 h

• lahgtna_eg_spk00001: 6.5 h

• lahgtna_eg_spk00002: 6.5 h

• lahgtna_eg_spk00003: 6.5 h

• lahgtna_eg_spk00004: 6.5 h

• lahgtna_eg_spk00005: 6.5 h

• lahgtna_eg_spk00006: 6.5 h

• masri_3shwa: 6.5 h

• masri_7akawi: 6.5 h

• OmniVoice's stock voice prompt (first 0-30% of a clip) and mask ratio 0-1; no instruction text.

Limitations

• The vowels in the training text are machine-made (Cohere-Speech-Tashkeel-2B) and cover about half of the words; the model learned that labelling, including its mistakes.
• lahgtna clips are 16 kHz upsampled to 24 kHz; masri is natively 24 kHz.
• Few speakers (the training data has 9); the other voices come from VoiceTut's own training.
• Not for impersonating a real person or deceiving listeners.

Acknowledgements

VoiceTut-TTS, OmniVoice by k2-fsa, the Higgs audio tokenizer v2, Rabe3/egyptian-arabic-tts-diacritized and its upstream sources, and Cohere-Speech-Tashkeel-2B.