kokoro-inno-clone-tuner

pip install inno-kokoro

Zero-shot voice tuner for Kokoro-82M.

Outputs base Kokoro compatible voice packs @ [510, 1, 256]. (write-up)

Samples

Same passage for every voice, enrolled from the references. LibriTTS-R speakers are dev-clean held out from training.

Usage

Code available on Github: remsky/inno-kokoro.

Integrated into Kokoro-FastAPI (v0.9.0+)

PyTorch, with kokoro

pip install "inno-kokoro[torch]"

from inno_kokoro.enroll import Tuner, enroll, read
import torch
from kokoro import KPipeline

tuner = Tuner(backend="torch") # caches model from HF on first use
pack, _ = enroll(*read("my_ref.wav"), tuner, return_pt=True) # [510, 1, 256]
torch.save(pack, "am_me.pt")

pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio

ONNX (no torch), with kokoro-onnx

pip install inno-kokoro kokoro-onnx

from inno_kokoro.enroll import Tuner, enroll, read
import numpy as np
from kokoro_onnx import Kokoro

tuner = Tuner(backend="onnx") # downloads model.safetensors + model.onnx on first use
pack, _ = enroll(*read("my_ref.wav"), tuner) # [510, 1, 256] float32 NumPy
np.save("am_me.npy", pack)

kokoro = Kokoro("kokoro-v1.0.onnx", "voices-v1.0.bin") # from the kokoro-onnx releases
wav, sr = kokoro.create("Hello from a tuned voice.", voice=pack, lang="en-us")

Notes:

• prefix names with stock accent+gender codes for use in other libraries: e.g af_, am_, bf_, bm_
• pitch ceiling is set automatically
  • enroll(..., fmax=180): can override when needed (archival sources, etc)

ONNX

model.onnx: the speaker encoder and style head for onnxruntime (fp32, 31.5 MB). Used by Tuner(backend="onnx") from inno-kokoro 0.4.0. Same packs as torch, about 2x faster on CPU.

Graph details

• input fbank [1, frames, 80] float32: kaldi log-mel of the first 30 s at 16 kHz (inno_kokoro/fbank.py)
• outputs embedding [1, 512] (unit, UniSpeech-SAT-sv space) and style [1, 256]
• opset 20, standard ops only, no DFT or custom ops
• matches torch within 5.4e-7 on 540 clips (parity check)
• exported with export_onnx.py (inno-kokoro tools/export_onnx.py at b442727)

CPU (best of 5 runs)

Architecture

A 5 to 30 s reference clip in, a stock-shaped Kokoro voice pack out. Everything loads from model.safetensors, no extra downloads.

Enrollment Architecture Diagram: reference audio to Kokoro-compatible voice pack. Architecture details

Enrollment embeds an input audio sample via the baked speaker encoder and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half.

• speaker encoder: a 6.6M-param WeSpeaker ResNet34 distilled from UniSpeech-SAT-sv embeddings
  • matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
• style head: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
• prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
  • Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
  • Packs more than 4 st from the reference's pitch are excluded.
  • Higher graded packs are weighted as preferred during selection.
• prosody head (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
  • fitted to deltas optimized through the frozen predictor. enroll(..., head=False) skips it.

Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.

Total model size @ fp16: ~24mb

Limits

• English trained (LibriTTS-R) currently
• Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
• Rythym is a best-effort match to the speaker.
  • Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
• Reference clip requirements:
  • 3 second minimum, up to a max of 30s
  • Reasonably free of audio artifacts
  • Single speaker

Training and Metrics

All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.

Trained via HF Jobs a10g-small; under $20 in GPU time:

v0.1.0 training notes

• Stage 1: mapped speaker embedding to timbre:
  • Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
  • [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
• Stage 2: the embedding alone missed texture and pitch; adjustments:
  • Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
  • [11k steps: dev cosine 0.934, DNSMOS 3.30]
• Stage 3: tuned the encoder side to match the blended predictors:
  • Predictor half fixed to the blended rows, only the decoder half trained.
  • [3k steps: dev cosine 0.932, DNSMOS 3.48]
• Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
  • WeSpeaker ResNet34 plus a projection, trained on 1 - cos(student, teacher) over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
  • [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
• Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
  • Mixed real public-domain archival broadcast audio into the distillation crops.
  • [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]

v0.2.0 training notes

• Stage 1: the blend under-predicted pitch range and sentence-final falls:
  • Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
  • Targeted F0 mean, spread, IQR, pause fraction and speech time.
  • Ridge head from reference F0 mean + spread to the fitted delta.
  • [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
• Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
  • Added VoxPopuli and Emilia speakers to the distillation crops.
  • [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
• Stage 3: the timbre half had only seen LibriTTS speakers:
  • Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
  • [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]

Zero-shot identity benchmark

Inno gets about a third of the normalized similarity range, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).

Benchmark Plot: Zero-shot speaker similarity comparison on LibriSpeech test-clean. Full benchmark results and methodology

LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers. Reference in, new sentence out, scored against the speaker's real recording.

• normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
• RTF: RTX 4060 Ti

StyleTTS2 uses a related architecture with an unfrozen decoder, a possible ceiling for this style space.

• StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
• OpenVoice v2: MeloTTS EN-US, tau 0.3.
• F5-TTS: 32 NFE, CFG 2.

v0.1 vs v0.2 shipped-weight comparison

Measured on the shipped weights through the stock KPipeline: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.

The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.

Licenses and credits

Adapter weights and code: Apache-2.0, except the baked speaker encoder (enc.* in model.safetensors), which is CC BY-SA 3.0 (see below)

• Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture)
• The speaker encoder is a WeSpeaker ResNet34 (code Apache-2.0)
  • initialised from the VoxCeleb-trained ResNet34-LM weights (CC BY 4.0)
  • distilled from microsoft/unispeech-sat-base-plus-sv
    • (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub); encoder weights under CC BY-SA 3.0
• Training data (no audio redistributed beyond the samples):
  • LibriTTS-R (CC BY 4.0), also the LibriTTS sample references
  • VoxPopuli (CC0)
  • Emilia-YODAS EN (CC BY 4.0)
• Sample references:
  • Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022
  • Jane Goodall from BBC Woman's Hour excerpt, BBC, CC BY 3.0
  • Vincent Price public domain from "The House On Haunted Hill"

Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.