中文Sign up

FluidInference/luxtts-coreml

🤗 Hugging Face sourcetext-to-speechapache-2.01.2 GBother✓ 30 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo FluidInference/luxtts-coreml ./model-folder
Needs a seeder →

LuxTTS CoreML

CoreML conversion of YatharthS/LuxTTS for on-device inference on Apple platforms.

LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip plus its transcript, and it speaks new text in that voice. It is based on ZipVoice (k2-fsa, Xiaomi) — a flow-matching Zipformer decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).

Models

Path Model Size Notes
gpu/TextEncoder.mlmodelc Text encoder (256 tokens) 9.5 MB fp16
gpu/FmDecoder.mlmodelc Flow-matching decoder, 1024-frame bucket 256 MB fp16, original graph — macOS GPU path
gpu-6bit/FmDecoder.mlmodelc 6-bit palettized decoder, 1024-frame bucket 115 MB full GPU speed, -0.45 dB, transcript-transparent
gpu-long/TextEncoder.mlmodelc Text encoder (512 tokens) 13 MB fp16
gpu-long/FmDecoder.mlmodelc Decoder, 2048-frame bucket 349 MB long-form macOS (~16.8 s per bucket)
ane/TextEncoder.mlmodelc Text encoder (256 tokens) 9.5 MB fp16
ane/FmDecoder.mlmodelc ANE-canonical decoder, 1024-frame bucket 313 MB 100% ANE placement — iPhone path
vocoder/Vocoder282.mlmodelc Vocos vocoder, 282 gen frames (~3.0 s) 33 MB ISTFT + 24k→48k resample + crossover in-graph
vocoder/Vocoder555.mlmodelc Vocos vocoder, 555 gen frames (~5.9 s) 33 MB matches the full 1024-frame bucket
tokens.txt / config.json Phoneme token table + model config — from upstream LuxTTS

Two decoder graphs — pick by compute unit

  • gpu/ — the original graph layout. Fastest on Mac GPU. Do not run it on the ANE: the seq-first rel-pos attention path loses precision there and produces corrupted audio.
  • ane/ — ANE-canonical rewrite (channels-first (1,C,1,S), 1×1-conv projections, constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback. This is the iPhone path; it is slower than the original graph on GPU, so keep both.

All decoder inputs/outputs are identical across graphs (t, x, text_condition, speech_condition, guidance_scale, padding_mask → v).

Performance (M5 Pro, macOS)

path fm step core RTFx (full bucket) steady footprint
gpu/ fp16, GPU 14.7 ms 92x (~5.9 s gen) 996 MB
gpu-6bit/, GPU 15.0 ms ~92x 419 MB
ane/ fp16, ANE 54.6 ms 27x 25.5 MB
  • Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~64 ms for 3 s of audio (~47x realtime).
  • Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts word-identical for all shipped variants.
  • Footprints are phys_footprint from a Swift host; the ANE graph keeps weights and activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.

Fixed-shape buckets

CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech (86.13 frames/s):

  • gpu/, ane/: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
  • gpu-long/: 2048 frames (~16.8 s generated), 512 text tokens
  • vocoder: fixed at 282 or 555 generated frames

Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.

Usage notes

The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT, 24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:

  • Phoneme tokenization against tokens.txt (espeak-compatible token set) — the frontend is phoneme-token based, not raw text.
  • Duration expansion of encoder output into the frame grid (upstream pad_labels keeps S+1 token rows; the pad slot's embedding fills remainder frames).
  • The 4-step anchor-Euler flow solver with t_shift = 0.5 and guidance_scale = 3.0.
  • Prompt trimming at a speech pause (VAD boundary), not a fixed duration — a prompt hard-cut mid-phrase makes the model elide sentence-initial words.
  • speed default 1.0 (upstream's generate() default of 1.3 clips sentence onsets).

Swift integration in FluidAudio is upcoming.

Known limitations

  • Fixed-shape buckets only (no enumerated/flexible shapes yet).
  • An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
  • The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers × 4 solver steps); transcripts are unaffected.

Attribution & license

  • LuxTTS by YatharthS — Apache-2.0.
  • Based on ZipVoice by k2-fsa (Xiaomi) — Apache-2.0.
  • Vocoder from LinaCodec (ysharma3501).
  • CoreML conversion by FluidInference; conversion source lives in the mobius repo (models/tts/zipvoice, PR #75).

References