nabra_sin_sad: Nabra-82M with ص and ض as their own sounds
This is Nabra-82M with ص kept apart from س and ض kept
apart from د. The voice (af_msa, female, Modern Standard Arabic, 24 kHz) is unchanged.
What changed
Nabra's phoneme cleanup strips the emphatic marks, so ص reached the model as the same token as س, and ض as the same token as د. As a result, plain س was often heard as ص and ض was often heard as د.
In this model:
ص and ض get their own tokens, using two free Kokoro ids:
Letter espeak-ng output New symbol Id ص s̪/sˤ26 ض dˤ27 Only the parts that decide which sound is said were trained, in two stages:
- The ص, ض,
sanddrows of the model's two token tables (PL-BERT word embeddings and the text encoder embedding). - The text encoder: 3 conv blocks and a BiLSTM, 5.6M of the 82M weights. In Arabic, ص also darkens the vowel next to it. Nabra had learned that a vowel after "s" is sometimes dark, and that is decided in the text encoder, so stage 1 alone left س sounding like ص in about 1 case in 6.
- The ص, ض,
The decoder (vocoder), the voicepack and the duration and pitch predictors are identical to Nabra-82M-v0.1. PL-BERT is identical apart from the 4 token rows. So the voice, the rhythm and the intonation are Nabra's.
Results
The tests use MSA sentences that were not used in training, transcribed by Whisper large-v3-turbo. This is a different Whisper from the one used in training (large-v3).
60 sentences (40 news-style sentences and 20 minimal pairs such as صار / سار):
| Nabra-82M-v0.1 | nabra_sin_sad v1 | nabra_sin_sad v2 (this version) | |
|---|---|---|---|
| س heard as ص | 23 of 98 | 16 of 98 | 1 of 98 |
| ص heard as س | 4 of 61 | 3 of 61 | 2 of 61 |
| ض heard as د | 6 of 38 | 5 of 38 | 0 of 38 |
| د heard correctly | 127 of 129 | 127 of 129 | 127 of 129 |
| Character error rate (all text) | 3.5% | 3.1% | 2.3% |
The 20 minimal pairs were also used to choose the checkpoint. On the 40 news-style sentences alone, v2 compares with v1 as follows:
- س heard as ص: 1 of 78 (v1: 11)
- ض heard as د: 0 of 27 (v1: 3)
- Character error rate: 2.6% (v1: 3.0%)
30 more sentences full of س (88 of them, in words like السيف, السفر, السنة and الحساب):
| nabra_sin_sad v1 | nabra_sin_sad v2 (this version) | |
|---|---|---|
| س heard as ص | 21 of 88 | 3 of 88 |
| Character error rate (all text) | 4.1% | 1.2% |
The vocoder adds random noise, so single counts can vary by one or two between runs.
Usage
Use the same pipeline as Nabra, with one extra step before the phoneme cleanup: ص and ض are mapped to their new symbols. Input should be diacritized (with tashkeel), as for Nabra.
pip install "kokoro @ git+https://github.com/Oddadmix/kokoro.git@main"
pip install torch soundfile huggingface_hub misaki
# the Arabic G2P front-end from the Nabra card
wget https://gist.githubusercontent.com/Oddadmix/dc699f7942a9516ce29d4842c7aed756/raw/827b541c892a862f9ef3b44006a6e27b100d1bdd/arabic_g2p.py
import re
import numpy as np, soundfile as sf, torch
from huggingface_hub import hf_hub_download
from arabic_g2p import EXTRA_SYMBOLS, clean_phonemes
from kokoro import KModel, KPipeline
from kokoro import pipeline as kpipeline_mod
REPO_ID = "Badawi001/nabra_sin_sad"
config = hf_hub_download(REPO_ID, "config.json")
weights = hf_hub_download(REPO_ID, "nabra_sin_sad.pth")
voice = torch.load(hf_hub_download(REPO_ID, "af_msa.pt"), map_location="cpu", weights_only=True)
SAD, DAD = "\ue100", "\ue101" # ص, ض
kmodel = KModel(repo_id=REPO_ID, config=config, model=weights, disable_complex=True).eval()
kmodel.vocab.update({**EXTRA_SYMBOLS, SAD: 26, DAD: 27}) # also listed in config.json
kpipeline_mod.LANG_CODES.setdefault("ar", "ar")
pipeline = KPipeline(lang_code="ar", repo_id=REPO_ID, model=kmodel)
espeak_g2p = pipeline.g2p
def g2p(text):
phonemes, tokens = espeak_g2p(text)
phonemes = re.sub("s[̪ˤ]+", SAD, phonemes) # ص: espeak writes s̪ or sˤ
phonemes = phonemes.replace("dˤ", DAD) # ض: espeak writes dˤ
phonemes = re.sub(f"([{SAD}{DAD}])\\1(?:\\1|ː)+", r"\1\1", phonemes) # no tripled consonants
return clean_phonemes(phonemes), tokens # cleanup must come after the mapping
pipeline.g2p = g2p
text = "صَارَ الصَّدِيقُ يَسِيرُ فِي السُّوقِ، وَأَوْضَحَ أَنَّ الضَّرَائِبَ تُسَدَّدُ فِي مَوْعِدِهَا."
wav = np.concatenate([a.detach().cpu().numpy() for _, _, a in pipeline(text, voice=voice, speed=1.0)])
sf.write("nabra_sin_sad.wav", wav, 24000)
If you leave out the mapping, the model still works, but ص and ض are read with the س and د tokens, as in the original Nabra.
ONNX
An ONNX export of the same weights is in onnx/. It runs on onnxruntime, without PyTorch.
| File | What it is |
|---|---|
onnx/nabra_sin_sad.onnx |
The model, fp32, opset 17, 326 MB |
onnx/voices.bin |
The af_msa voicepack as an npz file, shape [510, 1, 256] |
The inputs and outputs have the same names as kokoro-v1.0.onnx:
- Inputs:
input_idsint64[1, n],stylefloat[1, 256],speedfloat[1] - Outputs:
waveform(24 kHz) andduration
It was checked against the PyTorch model on 60 sentences:
- The predicted durations are identical in all 60.
- The audio differs only by the vocoder's random noise. The log-mel distance from PyTorch is the same as between two PyTorch runs (0.099).
- On a laptop CPU with 4 threads it runs at a real-time factor of 0.27; PyTorch runs at 0.29.
pip install onnxruntime soundfile huggingface_hub misaki phonemizer-fork espeakng-loader
# plus arabic_g2p.py, as above
import json, re
import numpy as np, onnxruntime as ort, soundfile as sf
from huggingface_hub import hf_hub_download
from misaki import espeak
from arabic_g2p import clean_phonemes
REPO_ID = "Badawi001/nabra_sin_sad"
vocab = json.load(open(hf_hub_download(REPO_ID, "config.json"), encoding="utf-8"))["vocab"]
session = ort.InferenceSession(hf_hub_download(REPO_ID, "onnx/nabra_sin_sad.onnx"))
voice = np.load(hf_hub_download(REPO_ID, "onnx/voices.bin"))["af_msa"] # [510, 1, 256]
SAD, DAD = "\ue100", "\ue101" # ص, ض
espeak_g2p = espeak.EspeakG2P(language="ar")
def phonemize(text):
phonemes, _ = espeak_g2p(text)
phonemes = re.sub("s[̪ˤ]+", SAD, phonemes) # ص: espeak writes s̪ or sˤ
phonemes = phonemes.replace("dˤ", DAD) # ض: espeak writes dˤ
phonemes = re.sub(f"([{SAD}{DAD}])\\1(?:\\1|ː)+", r"\1\1", phonemes) # no tripled consonants
return clean_phonemes(phonemes) # cleanup must come after the mapping
def speak(text, speed=1.0):
ids = [vocab[p] for p in phonemize(text) if p in vocab][:510]
audio, _ = session.run(None, {"input_ids": np.array([[0, *ids, 0]], dtype=np.int64),
"style": voice[len(ids) - 1],
"speed": np.array([speed], dtype=np.float32)})
return audio
text = "صَارَ الصَّدِيقُ يَسِيرُ فِي السُّوقِ، وَأَوْضَحَ أَنَّ الضَّرَائِبَ تُسَدَّدُ فِي مَوْعِدِهَا."
sf.write("nabra_sin_sad.wav", speak(text), 24000)
It also works with kokoro-onnx, using the phonemes from
phonemize() above:
from kokoro_onnx import Kokoro
kokoro = Kokoro("nabra_sin_sad.onnx", "voices.bin", vocab_config="config.json")
audio, sample_rate = kokoro.create(phonemize(text), voice="af_msa", is_phonemes=True)
Training
Both stages use the same teacher, loss and data:
| Teacher | Whisper large-v3 (frozen), listening to the generated audio |
| Loss | Cross-entropy of the correct transcript, plus a margin loss: the correct transcript must beat the same transcript with one ص/س or ض/د swapped (for example صار vs سار) by 4 nats. Plus an L2 pull toward the starting rows. |
| Data | About 800 MSA news-style sentences, diacritized with CATT and phonemized with espeak-ng (text only; the audio is generated by the model itself) |
The stages differ in what they train:
| Stage 1 | Stage 2 | |
|---|---|---|
| Trainable | 4 rows (ids 26, 27, s=61, d=46) of bert.embeddings.word_embeddings (128-d) and text_encoder.embedding (512-d). Rows 26 and 27 start as copies of s and d. |
The text encoder's conv blocks and BiLSTM, plus the same 4 rows of text_encoder.embedding. Starts from stage 1. |
| Frozen | Everything else, including the voicepack | Everything else, including PL-BERT, the predictors, the decoder and the voicepack |
| Extra loss | None | A keep loss: the text encoder's output must stay equal to stage 1's, except within 2 tokens of س ص د ض (weight 0.1 there), so the other sounds don't change |
| Schedule | Adam, learning rate 6e-3, 50 steps of 4 sentences | Adam, learning rate 3e-4 for the layers and 1e-3 for the rows, 400 steps of 4 sentences |
In both stages, the checkpoint was chosen on a separate set of 20 minimal-pair sentences.
Limitations
- A few س are still heard as ص (see Results).
- ط/ت and ظ/ذ are still merged, as in Nabra.
- Diacritized input is still required, MSA only, single voice: the same limits as Nabra-82M.
Credits
The base model is oddadmix/Nabra-82M-v0.1, a fine-tune of hexgrad/Kokoro-82M (StyleTTS2 / ISTFTNet). Diacritization uses CATT.