Vui — Streaming Conversational Voice Assistant

Pronounced "vooey" (rhymes with Louie) · by fluxions.ai

👉 Full code, install, docs, and the streaming voice assistant: github.com/fluxions-ai/vui

📖 Launch blog post — design notes, demos, and what's next.

Vui Nano is a small, context-aware text-to-speech model trained on real conversations: 219M active parameters (305M total), Apache 2.0, with voice cloning, real-time streaming, and a dependency-free C build that runs on CPU.

Most TTS models synthesise one utterance in isolation. Vui Nano generates each reply inside the conversation: the whole dialogue so far — your text and the actual audio of your turn — lives in the KV cache it decodes from, across a ~6-minute context. It was trained on two-speaker dialogue with an explicit speaker-change token, so it carries prosody across turns and produces the things real speech has and read-aloud corpora don't: breaths, laughter, hesitations, and overlap.

The handful of other open models that condition on dialogue acoustics this way are an order of magnitude larger and GPU-only. Vui Nano does it at 219M active parameters, and the C build runs it on a CPU with no Python, PyTorch, or ONNX at runtime.

It ships inside Vui, a real-time voice assistant: speak into your mic, the model transcribes, runs a local LLM, and streams a TTS reply back — all from a single Python server.

Features

• Vui Nano (219M active, 305M total) — a small, context-aware TTS model: Llama-style decoder + RQ-Transformer head over the Qwen3-TTS-12Hz codec, Apache 2.0
• Conversation-conditioned generation — replies are decoded from a KV cache holding the whole dialogue, including the audio of your turn (~6-minute context)
• CPU inference, zero dependencies — a pure-C engine: one binary plus one weight file, no Python, PyTorch, or ONNX at runtime; supports voice cloning and streaming playback
• Real-time voice loop — WebRTC + WebSocket pipeline (ASR → LLM → TTS) with a browser UI, VAD-driven turn taking, speculative LLM prefill while you're still speaking, sentence-level TTS chunking with backpressure
• Barge-in — start talking mid-reply, the model cancels and listens
• Streaming TTS — ~9× realtime on a 4090, bf16 inference, CUDA graphs
• OpenAI Realtime API compatible — drop-in ws://…/v1/realtime for clients written against OpenAI's spec (docs/realtime-api.md)
• One-shot voice-note REST endpoint — POST /v1/voice-note runs the whole ASR → LLM → TTS pipeline in a single HTTP call (audio in, JSON out)
• Standalone TTS demo — demo.py Gradio playground for the model on its own
• Voice cloning — upload an audio sample to clone any speaker; 4 fine-tuned presets shipped (maeve, abraham, rhian, harry)
• SQ / WPS conditioning — bias generation on six speech-quality channels and words-per-second
• Hot-swap models — pick Ollama LLM and ASR backend live from the UI
• Pluggable ASR — faster-whisper (GPU) or Moonshine (CPU streaming, ONNX)
• Pluggable LLM backends — Ollama, vLLM, any OpenAI-compatible endpoint
• Memories — assistant remembers facts about you across sessions
• Thoughts stream — parallel LLM routes voice intent to ~10 tools (memory ops, task control, delegation) without a wake-word grammar; pluggable for your own local tools
• Optional Claude task server — sidecar agent that handles slow/agentic work (Gmail, Calendar, Drive, Slack, web search) via your existing Claude Code MCPs
• Apple Silicon support — MLX backend (WIP)
• Mobile-ready — documented cloudflared and Tailscale paths for phone access with mic over HTTPS
• Docker compose — one file brings up the full stack
• OpenClaw integration — point OpenClaw's openai realtime provider at Vui for a fully-local voice front-end

Install (one-liner)

curl -fsSL https://install.fluxions.ai | bash

Clones into ~/vui, auto-detects Docker vs. native, installs deps (uv, Ollama, ffmpeg, Claude Code CLI), pulls the Ollama LLM, and launches the stack on http://localhost:8080.

Full Docker compose / native install, mobile setup, configuration, ASR options, and the Claude task server — all in the GitHub README.

TTS demo on its own

git clone https://github.com/fluxions-ai/vui
cd vui
uv sync
python demo.py                                          # Gradio UI — upload your own voice prompt
python demo.py --render --prompt prompts/abraham.wav    # CLI render with a preset voice

The Vui checkpoint and Qwen codec download automatically from this repo on first run.

Preset voices

More personalities coming soon! Got a voice or character you'd like to hear? Open an issue or let us know on Discord.

Python API

from vui.engine import Engine, GenConfig

engine = Engine()  # vui-nano-1.1 by default; Engine("vui-190k") / Engine("vui-nano") for the others
with engine.new_row() as row:
    codes, audio = row.render(   # render() returns (codes (T,Q), audio (1,1,S))
        "So [breath] the thing about this is, it's not what you'd expect, right?",
        GenConfig(temperature=0.7),
    )

Clone a voice

Cloning is a prefill: hand the model Segment(text, codes) — a reference transcript plus its encoded audio — and everything rendered afterwards follows that speaker.

import torch
from julius.resample import resample_frac
from torchcodec.decoders import AudioDecoder
from torchcodec.encoders import AudioEncoder

from vui.engine import Engine, GenConfig, Segment
from vui.inference import asr
from vui.qwen_codec import SAMPLE_RATE as SR  # 24 kHz
from vui.qwen_codec import QwenCodecEncoder

engine = Engine()
dev = "cuda" if torch.cuda.is_available() else "cpu"

wav_16k = AudioDecoder("prompts/abraham.wav", sample_rate=16000, num_channels=1) \
    .get_all_samples().data.squeeze(0)

codec_enc = QwenCodecEncoder.from_pretrained().to(dev).float().eval()
with torch.inference_mode():
    codes = codec_enc.encode(
        resample_frac(wav_16k.unsqueeze(0), 16000, SR).float().to(dev).unsqueeze(0)
    )
prompt_codes = codes[0, : engine.Q].T.long()   # (T, Q)
prompt_text = asr(wav_16k)                     # must match the audio word-for-word

with engine.new_row() as row:
    row.prefill([Segment(prompt_text, prompt_codes)])
    _, audio = row.render(
        "So [breath] the thing about this is, it's not what you'd expect.",
        GenConfig(temperature=0.7, max_secs=10),
    )

AudioEncoder(audio.squeeze().cpu().float().unsqueeze(0), sample_rate=SR).to_file("out.wav")

row.rewind() returns the KV cache to end-of-prompt, so you can render many lines in one voice without re-encoding the reference. From the CLI, python demo.py --render --prompt your_voice.wav --text "..." does the same thing in one line.

Clone a voice on CPU

The C build clones too — no Python at runtime:

cd cpu
python export_full.py vui-nano-1.1.safetensors vui_full.bin      # one-time
gcc -O3 -march=native -ffast-math -fopenmp -o vui_tts vui_tts.c -lm -lopenblas

python prepare_prompt.py /path/to/your_voice.wav prompt_cache.bin
OMP_NUM_THREADS=4 ./vui_tts vui_full.bin --kv-cache prompt_cache.bin \
    --text "Hello from a CPU." --output out.wav

Tip: try turning repetition penalty off. GenConfig defaults rep_penalty=1.1 to break long silence/filler loops, but it can flatten prosody and distort natural repetition. Setting it to 0 (anything <= 1.0 disables the penalty path) often gives more natural-sounding output — worth trying if generations sound stilted or over-corrected.

For long voice prompts (>15s) you need proper multi-segment chunking — vui.prompt_utils.build_prompt_segments does ASR + forced alignment + sentence-boundary splits at ~10s targets so the model keeps its speaker conditioning across the full reference. Full Python guide covering chunked prompts, streaming, continuous batching, codes-only decode, and the MLX path: docs/python-api.md.

Vui Nano

A small autoregressive LM over the Qwen3-TTS speech codec — 219M active parameters, 305M total — and the first in the Vui model family. The codec and speaker encoder are reused from Alibaba's Qwen3-TTS-12Hz-0.6B-Base;

• 218,623,489 active parameters of 305,356,033 total (71.6%) — the remaining 86,732,544 are embedding lookup tables that cost zero FLOPs. Llama-style decoder + RQ-Transformer head — 768 dim, 22 layers, 8 heads
• Codec: Qwen3-TTS-Tokenizer-12Hz — 16 codebooks of 2048 entries at 12.5 Hz, 24 kHz audio (decoded), pure-PyTorch reimplementation in src/vui/qwen_codec.py
• Speaker encoder: ECAPA-TDNN from Qwen3-TTS-12Hz-0.6B-Base (8.9M params, 1024-dim) — used at training time to embed reference speakers
• Output: 24 kHz audio, bf16 inference (611 MB on disk), ~9× realtime streaming on a 4090
• Conversation context: ~6 minutes (4500 frames at 12.5 Hz). A turn enters the KV cache as text [SC] codes — the user's words and their audio — so generation is conditioned on the dialogue so far, not just the sentence being spoken
• Trained on dialogue: two-speaker alternating turns with an explicit [SC] speaker-change token, plus 21 paralinguistic tokens — [breath], [laugh], [hesitate], [sigh], [overlap] …
• License: Apache 2.0, weights included — commercial use permitted

Checkpoints

All three share the architecture above. Engine(), demo.py and the streaming server default to vui-nano-1.1; pass a name from this table, a filename from this repo, or a local path to pick another.

Voice prompts. prompts/<voice>.safetensors holds codec codes (checkpoint-agnostic — the Python engine and the streaming server prefill from these) plus a baked cond_bias / spk_token_emb pair that is checkpoint-specific, with the exact transcript in the file's metadata. Only the cpu/ C engine and the MLX/iOS prebake read the baked pair, so take prompts from the folder baked for your checkpoint: prompts/vui-nano-1.1/ for 1.1, prompts/ for vui-nano. scripts/build_prompts.py in the repo regenerates a folder for any checkpoint.

Where the parameters go

The compute-weighted picture inverts. Per 80 ms audio frame the backbone runs once, but the RQ transformer runs 15 times — once per quantizer after the first:

So the 35M-parameter RQ head — 11.6% of the weights — is three-quarters of the arithmetic, which is why n_codebooks moves latency so much: dropping 16 to 10 cuts per-frame MACs by 31%, 16 to 8 by 42%.

One realtime stream is ~17.8 GFLOP/s. Against a 4090's dense bf16 throughput that is well under 1% utilisation — Vui Nano is latency- and memory-bound, not compute-bound.

Voices & voice cloning

The model can clone arbitrary voices — upload a sample in the demo UI (or drop a .wav into prompts/) and it will follow that speaker. Cloned voices won't sound as good as the four fine-tuned voices (maeve, abraham, rhian, harry) shipped in prompts/ — the released checkpoint has been fine-tuned on those four, so they're the highest-quality output the model can produce. Arbitrary clones work but expect lower naturalness, more drift, and some bias toward the fine-tuned speakers' prosody.

For best results: voice-prompt transcript must match the audio word-for-word, aim for 30 seconds or more of clean source audio (6-minute context window), and remember garbage in = garbage out. Full guide on voice prompts, supported tags ([breath], [laugh], [sigh] …), punctuation rules, and phonetic spelling for numbers/dates/units: docs/prompting.md.

If you need a checkpoint tuned to a specific voice for a legitimate use case (audiobooks, accessibility, game characters, dubbing of consenting performers, internal tooling), get in touch via fluxions.ai — we can train, license, or host one for you.

Hardware

Streaming server and demo.py both run on either:

• NVIDIA GPU + Linux — ~12 GB VRAM for the full stack (TTS + ASR + Ollama LLM, 4090 / H100 tested), drops to ~8 GB if you switch to a moonshine.* (CPU) ASR backend. CUDA 12.x, flash-attn installed.
• Apple Silicon Mac — M1/M2/M3/M4, MLX backend (auto-detected, no flash-attn required).

Full breakdown — measured per-component VRAM, ASR latency/VRAM per backend, KV-cache math, and tuning levers — is in docs/memory-budget.md.

Tip: drop n_codebooks for faster TTS on smaller GPUs. The RQ-Transformer head decodes 16 RVQ codebook levels per audio frame by default. Dropping the Codebooks slider in the UI (or n_codebooks in DEFAULT_SETTINGS) to ~10 gives noticeably faster decode and lower VRAM at the cost of some stability — occasional artefacts, more sensitivity to hard prompts. Below 8 quality drops sharply. 0 means "use all 16".

Responsible use

Vui generates speech that can sound convincingly human. By using this model — directly, through the streaming server, or through the realtime API — you agree to the following:

We explicitly prohibit:

• Fraud — generating speech to deceive others for financial gain or to obtain something you would not otherwise be entitled to (scam calls, voice-auth bypass, etc.).
• Misinformation or deception — fake news, fraudulent calls, deepfakes intended to mislead, synthetic media presented as authentic recordings of real people.
• Harassment, defamation, or abuse — generating speech that targets, threatens, or harms others, including non-consensual sexual content.
• Illegal activity — anything unlawful in the jurisdiction where the model is run or its output is distributed.

You are responsible for what you generate. The released checkpoint is fine-tuned to a curated voice set in part to make these misuses harder, but it is not a substitute for your own judgment. If you build a product on top of Vui, build in consent flows, content provenance (e.g. C2PA), and abuse reporting.

We are not responsible for misuse, and we strongly condemn unethical applications of this technology.

Attributions

• Qwen3-TTS-Tokenizer — Alibaba
• Whisper — OpenAI
• faster-whisper
• Moonshine — Useful Sensors (CPU-streaming ASR option)
• Silero VAD
• aiortc
• Ollama

License

Apache 2.0 — applies to the code in the GitHub repo and the released model weights. The Qwen3-TTS-Tokenizer-12Hz codec and Qwen3-TTS-12Hz-0.6B-Base speaker encoder are © Alibaba and licensed under the terms in their respective Hugging Face repos.

Citation

@software{vui_2026,
  author  = {Coultas Blum, Harry},
  title   = {Vui: Streaming Conversational Text-to-Speech},
  url     = {https://github.com/fluxions-ai/vui},
  version = {1.0.0},
  year    = {2026}
}