Byrne-Speech (female voice)

Model will be ungated for open download once I am done with the base.

Tiny (~12M) English TTS. From scratch, trained on one 12 GB GPU.

text → characters → acoustic transformer → 80-bin mel @ 24 kHz → HiFi-GAN → waveform

Architecture

Non-autoregressive FastSpeech2-style transformer, but the blocks are SpikeWhale-v2 family primitives, not vanilla FastSpeech2:

• RMSNorm everywhere (including the variance predictors). No LayerNorm.
• RoPE + QK-Norm attention - rotary PE with query/key norm (Gemma2 / OLMo2-style logit stability). Replaces sinusoidal PE + vanilla attention.
• Conv-SwiGLU FFN - gated silu(gate) * up → down with a depthwise conv. Keeps FastSpeech's local context.
• DERF-gated value-embedding residual (DERFValueGate) - every block re-injects the character embedding through an erf gate (use_value_embed: true). Gate init is bias=-2 so it actually opens and contributes during training.
• HRM refinement (HRMRefinement) - iterative polish on the mel decoder (hrm_refine_steps: 3, hrm_refine_dim: 192). Gate-init is fixed - only the scalar gate is zeroed, not the output projection - so it gets gradient and learns a real residual.

Pipeline: char embedding → 3 encoder FFT blocks → variance adaptor (duration / pitch / energy, 256-bin pitch & energy) → length regulation with forced-aligned durations → 3 decoder FFT blocks → HRM refinement → 80-bin mel. Dims: hidden 256, 2 heads, FFN 320, conv kernel 3.

Active refinement (v2). Earlier checkpoints shipped with the two gated add-ons inert - over-conservative init left HRM refine and the DERF value-embed gate without gradient, so they sat at zero and all quality came from the core blocks. This checkpoint zeros only one side of each gate: HRM trains to a real strength, DERF opens, and the mel gets sharper - more spectral detail, less buzz.

Vocoder: HiFi-GAN generator with Snake activation (periodic inductive bias), upsample 256× to 24 kHz, MPD + MSD discriminators, then fine-tuned on the acoustic model's predicted mels so train/inference match.

Voice is clear and naturally paced (forced-aligner durations, not a uniform heuristic), then distilled from a VITS teacher to sharpen spectral detail. Still 12M.

Usage

pip install -r requirements.txt
python -m src.infer --text "Hello, this is the Byrne Speech model." \
    --config config.yaml --acoustic acoustic.pt --vocoder vocoder.pt --out out.wav

Or in Python:

import torch
from src.utils import load_config
from src.infer import load_acoustic, load_vocoder, synthesize
cfg = load_config("config.yaml")
dev = "cuda" if torch.cuda.is_available() else "cpu"
ac = load_acoustic("acoustic.pt", cfg, dev)
voc = load_vocoder("vocoder.pt", cfg, dev)
wav = synthesize("Text to speak.", ac, voc, cfg, dev)   # float32 @ 24 kHz

Inference is light - torch, numpy, soundfile, pyyaml, inflect, scipy. No phonemizer, no g2p, no torchaudio. Fine on CPU.

Samples

See samples/ - sample_1_fox.wav, sample_2_welcome.wav, sample_3_weather.wav, sample_4_thanks.wav. Live demo is in the companion Space.

Fine-tuning to your own voice

A few hours of clean single-speaker audio is enough. See FINETUNE.md (--finetune --lr 1e-4).

Notes & provenance

• Character input (no g2p). English text goes in as-is.
• Durations are forced-aligned (torchaudio MMS).
• Distilled from kakao-enterprise/vits-ljs (VITS on public-domain LJSpeech). Quality is a 12M model + mel L1, so a slight soft/even texture is by design.
• A male voice is in progress. Later update.

Small model, from scratch. Intelligible, fast, tiny.

Citation

@misc{byrne_speech_2026,
  title        = {Byrne-Speech: a tiny (~12M) from-scratch English text-to-speech model},
  author       = {Byrne, Dean},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Speech}}
}

Built on public-domain LJSpeech, distilled from kakao-enterprise/vits-ljs.