加载中…
加载中…
Pronounced "vooey" (rhymes with Louie) · by fluxions.ai
👉 Full code, install, docs, and the streaming voice assistant: github.com/fluxions-ai/vui
📖 Launch blog post — design notes, demos, and what's next.
Vui Nano is a small, context-aware text-to-speech model trained on real conversations: 219M active parameters (305M total), Apache 2.0, with voice cloning, real-time streaming, and a dependency-free C build that runs on CPU.
Most TTS models synthesise one utterance in isolation. Vui Nano generates each reply inside the conversation: the whole dialogue so far — your text and the actual audio of your turn — lives in the KV cache it decodes from, across a ~6-minute context. It was trained on two-speaker dialogue with an explicit speaker-change token, so it carries prosody across turns and produces the things real speech has and read-aloud corpora don't: breaths, laughter, hesitations, and overlap.
The handful of other open models that condition on dialogue acoustics this way are an order of magnitude larger and GPU-only. Vui Nano does it at 219M active parameters, and the C build runs it on a CPU with no Python, PyTorch, or ONNX at runtime.
It ships inside Vui, a real-time voice assistant: speak into your mic, the model transcribes, runs a local LLM, and streams a TTS reply back — all from a single Python server.
ws://…/v1/realtime for clients written against OpenAI's spec (docs/realtime-api.md)POST /v1/voice-note runs the whole ASR → LLM → TTS pipeline in a single HTTP call (audio in, JSON out)demo.py Gradio playground for the model on its ownmaeve, abraham, rhian, harry)openai realtime provider at Vui for a fully-local voice front-endcurl -fsSL https://install.fluxions.ai | bash
Clones into ~/vui, auto-detects Docker vs. native, installs deps (uv, Ollama, ffmpeg, Claude Code CLI), pulls the Ollama LLM, and launches the stack on http://localhost:8080.
Full Docker compose / native install, mobile setup, configuration, ASR options, and the Claude task server — all in the GitHub README.
git clone https://github.com/fluxions-ai/vui
cd vui
uv sync
python demo.py # Gradio UI — upload your own voice prompt
python demo.py --render --prompt prompts/abraham.wav # CLI render with a preset voice
The Vui checkpoint and Qwen codec download automatically from this repo on first run.
| Voice | Description |
|---|---|
maeve |
Recommended Default — Female Irish accent, beautiful but may be hard for non-UK listeners |
abraham |
British, well-spoken, exciting energy and personality — conscientious, good at emotionally difficult subjects |
rhian |
More traditional British accent, slightly hesitant speaking style |
harry |
British male accent, mumbly |
More personalities coming soon! Got a voice or character you'd like to hear? Open an issue or let us know on Discord.
from vui.engine import Engine, GenConfig
engine = Engine() # vui-nano-1.1 by default; Engine("vui-190k") / Engine("vui-nano") for the others
with engine.new_row() as row:
codes, audio = row.render( # render() returns (codes (T,Q), audio (1,1,S))
"So [breath] the thing about this is, it's not what you'd expect, right?",
GenConfig(temperature=0.7),
)
Cloning is a prefill: hand the model Segment(text, codes) — a reference transcript plus its encoded audio — and everything rendered afterwards follows that speaker.
import torch
from julius.resample import resample_frac
from torchcodec.decoders import AudioDecoder
from torchcodec.encoders import AudioEncoder
from vui.engine import Engine, GenConfig, Segment
from vui.inference import asr
from vui.qwen_codec import SAMPLE_RATE as SR # 24 kHz
from vui.qwen_codec import QwenCodecEncoder
engine = Engine()
dev = "cuda" if torch.cuda.is_available() else "cpu"
wav_16k = AudioDecoder("prompts/abraham.wav", sample_rate=16000, num_channels=1) \
.get_all_samples().data.squeeze(0)
codec_enc = QwenCodecEncoder.from_pretrained().to(dev).float().eval()
with torch.inference_mode():
codes = codec_enc.encode(
resample_frac(wav_16k.unsqueeze(0), 16000, SR).float().to(dev).unsqueeze(0)
)
prompt_codes = codes[0, : engine.Q].T.long() # (T, Q)
prompt_text = asr(wav_16k) # must match the audio word-for-word
with engine.new_row() as row:
row.prefill([Segment(prompt_text, prompt_codes)])
_, audio = row.render(
"So [breath] the thing about this is, it's not what you'd expect.",
GenConfig(temperature=0.7, max_secs=10),
)
AudioEncoder(audio.squeeze().cpu().float().unsqueeze(0), sample_rate=SR).to_file("out.wav")
row.rewind() returns the KV cache to end-of-prompt, so you can render many lines in one voice without re-encoding the reference. From the CLI, python demo.py --render --prompt your_voice.wav --text "..." does the same thing in one line.
The C build clones too — no Python at runtime:
cd cpu
python export_full.py vui-nano-1.1.safetensors vui_full.bin # one-time
gcc -O3 -march=native -ffast-math -fopenmp -o vui_tts vui_tts.c -lm -lopenblas
python prepare_prompt.py /path/to/your_voice.wav prompt_cache.bin
OMP_NUM_THREADS=4 ./vui_tts vui_full.bin --kv-cache prompt_cache.bin \
--text "Hello from a CPU." --output out.wav
Tip: try turning repetition penalty off. GenConfig defaults rep_penalty=1.1 to break long silence/filler loops, but it can flatten prosody and distort natural repetition. Setting it to 0 (anything <= 1.0 disables the penalty path) often gives more natural-sounding output — worth trying if generations sound stilted or over-corrected.
For long voice prompts (>15s) you need proper multi-segment chunking — vui.prompt_utils.build_prompt_segments does ASR + forced alignment + sentence-boundary splits at ~10s targets so the model keeps its speaker conditioning across the full reference. Full Python guide covering chunked prompts, streaming, continuous batching, codes-only decode, and the MLX path: docs/python-api.md.
A small autoregressive LM over the Qwen3-TTS speech codec — 219M active parameters, 305M total — and the first in the Vui model family. The codec and speaker encoder are reused from Alibaba's Qwen3-TTS-12Hz-0.6B-Base;
src/vui/qwen_codec.pyQwen3-TTS-12Hz-0.6B-Base (8.9M params, 1024-dim) — used at training time to embed reference speakerstext [SC] codes — the user's words and their audio — so generation is conditioned on the dialogue so far, not just the sentence being spoken[SC] speaker-change token, plus 21 paralinguistic tokens — [breath], [laugh], [hesitate], [sigh], [overlap] …All three share the architecture above. Engine(), demo.py and the streaming server default to vui-nano-1.1; pass a name from this table, a filename from this repo, or a local path to pick another.
| Name | What it is | Paired eval (12 lines, abraham prompt, moonshine ASR, 4090) |
|---|---|---|
vui-nano-1.1 (default) |
RL-tuned from vui-190k — the checkpoint that has served the production API since 2026-08-18. Same weights layout; 6-channel SQ conditioning. Over 2400 production renders: catastrophic failures 1.54% → 0.71%, mean WER 5.2% → 3.0%. No babble gate needed. |
WER 2.9%, 0 lines over 10%, 9.1× realtime |
vui-190k |
Run 3hggswum step 190k — the 1.0.x default and the base of 1.1. 7-channel SQ conditioning; babble_probe-190k.pt targets this checkpoint. |
WER 9.4%, 5 lines over 10%, 8.7× realtime |
vui-nano |
The original 1.0 release checkpoint (6-channel SQ). Kept for reproducibility. | — |
Voice prompts. prompts/<voice>.safetensors holds codec codes (checkpoint-agnostic — the Python engine and the streaming server prefill from these) plus a baked cond_bias / spk_token_emb pair that is checkpoint-specific, with the exact transcript in the file's metadata. Only the cpu/ C engine and the MLX/iOS prebake read the baked pair, so take prompts from the folder baked for your checkpoint: prompts/vui-nano-1.1/ for 1.1, prompts/ for vui-nano. scripts/build_prompts.py in the repo regenerates a folder for any checkpoint.
| Component | Params | Share | Lookup only |
|---|---|---|---|
| Backbone — 22 layers | 155,748,096 | 51.0% | |
| Text embedding (49,429 x 768) | 37,961,472 | 12.4% | Y |
| RQ transformer — 5 layers | 35,397,120 | 11.6% | |
| Audio embedding (16 x 2048) | 25,165,824 | 8.2% | Y |
| RQ code embedding (15 x 2048) | 23,592,960 | 7.7% | Y |
RQ output heads head_W (15 x 2048 x 768) |
23,592,960 | 7.7% | |
| SQ / WPS / speaker projectors | 2,310,912 | 0.8% | |
codec_head + eos_head + final norm |
1,574,401 | 0.5% | |
| RQ position embedding (16 x 768) | 12,288 | 0.0% | Y |
| Total | 305,356,033 | ||
| Lookup only (zero FLOPs) | 86,732,544 | 28.4% | |
| Active in the compute path | 218,623,489 | 71.6% |
The compute-weighted picture inverts. Per 80 ms audio frame the backbone runs once, but the RQ transformer runs 15 times — once per quantizer after the first:
| Stage | MACs / frame | Share |
|---|---|---|
| RQ transformer — 5 layers x 15 steps | 530,841,600 | 74.6% |
| Backbone — 22 layers x 1 step | 155,713,536 | 21.9% |
head_W x 15 |
23,592,960 | 3.3% |
codec_head x 1 |
1,572,864 | 0.2% |
| Total | 711,720,960 |
So the 35M-parameter RQ head — 11.6% of the weights — is three-quarters of the arithmetic, which is why n_codebooks moves latency so much: dropping 16 to 10 cuts per-frame MACs by 31%, 16 to 8 by 42%.
One realtime stream is ~17.8 GFLOP/s. Against a 4090's dense bf16 throughput that is well under 1% utilisation — Vui Nano is latency- and memory-bound, not compute-bound.
The model can clone arbitrary voices — upload a sample in the demo UI (or drop a .wav into prompts/) and it will follow that speaker. Cloned voices won't sound as good as the four fine-tuned voices (maeve, abraham, rhian, harry) shipped in prompts/ — the released checkpoint has been fine-tuned on those four, so they're the highest-quality output the model can produce. Arbitrary clones work but expect lower naturalness, more drift, and some bias toward the fine-tuned speakers' prosody.
For best results: voice-prompt transcript must match the audio word-for-word, aim for 30 seconds or more of clean source audio (6-minute context window), and remember garbage in = garbage out. Full guide on voice prompts, supported tags ([breath], [laugh], [sigh] …), punctuation rules, and phonetic spelling for numbers/dates/units: docs/prompting.md.
If you need a checkpoint tuned to a specific voice for a legitimate use case (audiobooks, accessibility, game characters, dubbing of consenting performers, internal tooling), get in touch via fluxions.ai — we can train, license, or host one for you.
Streaming server and demo.py both run on either:
moonshine.* (CPU) ASR backend. CUDA 12.x, flash-attn installed.Full breakdown — measured per-component VRAM, ASR latency/VRAM per backend, KV-cache math, and tuning levers — is in docs/memory-budget.md.
Tip: drop n_codebooks for faster TTS on smaller GPUs. The RQ-Transformer head decodes 16 RVQ codebook levels per audio frame by default. Dropping the Codebooks slider in the UI (or n_codebooks in DEFAULT_SETTINGS) to ~10 gives noticeably faster decode and lower VRAM at the cost of some stability — occasional artefacts, more sensitivity to hard prompts. Below 8 quality drops sharply. 0 means "use all 16".
Vui generates speech that can sound convincingly human. By using this model — directly, through the streaming server, or through the realtime API — you agree to the following:
We explicitly prohibit:
You are responsible for what you generate. The released checkpoint is fine-tuned to a curated voice set in part to make these misuses harder, but it is not a substitute for your own judgment. If you build a product on top of Vui, build in consent flows, content provenance (e.g. C2PA), and abuse reporting.
We are not responsible for misuse, and we strongly condemn unethical applications of this technology.
Apache 2.0 — applies to the code in the GitHub repo and the released model weights. The Qwen3-TTS-Tokenizer-12Hz codec and Qwen3-TTS-12Hz-0.6B-Base speaker encoder are © Alibaba and licensed under the terms in their respective Hugging Face repos.
@software{vui_2026,
author = {Coultas Blum, Harry},
title = {Vui: Streaming Conversational Text-to-Speech},
url = {https://github.com/fluxions-ai/vui},
version = {1.0.0},
year = {2026}
}