No GPU? Try it through the hosted API: sign up and get 3 hours of free voice chat (OpenAI Realtime-compatible, docs).
🆕 Smaller GPU? Sophon-Realtime-4B-Tag (v1.4.0) runs this build with a 4B brain on one 32 GB GPU: keeps the turn-taking and sound tags, knows much less. All builds
Version 1.2.0 (2026-11-13) · changelog · ▶ v1.1 demo video
Code and one-command deployment: github.com/adventists-ai/sophon-realtime/tree/main/tag · Built on DuplexJev · Voice: tag-tts
The Tag build of Sophon-Realtime: it decides how to say it, and runs on one GPU. The 32B thinker listens to you, decides when it is its turn, and writes the reply together with inline sound tags: a laugh after a joke, a sigh for bad news, stress on the word that matters. A streaming voice (tag-tts) performs those tags while the reply is still being written. Tags are never read aloud and are removed from the text stream.
The whole system is one vLLM-Omni model: thinker → tag-tts talker → flow + HiFT vocoder, on a single 80 GB GPU.
Three builds share one brain (the same 32B model listens, decides turns and writes the reply). They differ only in how the reply becomes sound. Full comparison: github.com/adventists-ai/sophon-realtime.
| Tag (recommended) | Flash | E2E | |
|---|---|---|---|
| How it speaks | an expressive speech model performs the sound tags | a lighter, faster speech model speaks the tagged reply | a compact talker (about 350M parameters) generates continuous voice vectors directly, no speech tokens |
| NVV-SuperBench (en / zh) | 0.80 / 0.86 | 0.63 / 0.77 | 0.75 / 0.85 |
| First audio, one user | ~0.6 s | 0.31 s | ~0.25 s |
| Users per 80 GB GPU (recommended) | 16 | 40 | 16 |
| GPUs | 1 | 1 | 1 |
| Weights | Sophon-Realtime-32B-Tag | Sophon-Realtime-32B-Flash | Sophon-Realtime-32B-E2E |
Pick Tag if you want the voice to act; pick Flash if users per GPU matter most; pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.
Benchmark methods and full comparison tables: v1 family home page. To just try it: talk to it in the ModelScope Studio.
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh # Python 3.12 venv, vLLM 0.30, vLLM-Omni 0.30, the plugin, and these weights (~42 GB)
GPU=0 PORT=8000 bash start.sh # first start 15–20 min (kernels + TensorRT engines), later 5–8 min
python client_example.py --wav my_question.wav --lang en --voice ryan_en
ws://HOST:8000/v1/realtime (full duplex: turn-taking, barge-in, backchannel) and POST /v1/chat/completions
(audio in → text + audio stream).chelsie (default; chelsie_zh / chelsie_en picked by language), vivian_zh, ryan_en, brand_zh, brand_en.Give it a short clip of any voice and talk to Sophon in that voice. No fine-tuning, no restart.
cp alice.wav model/voices/alice.wav # 3–10 s of clean speech from one speaker (wav or flac, any sample rate)
python client_example.py --wav my_question.wav --lang en --voice alice
"voice": "alice" in /v1/chat/completions or in session.update on /v1/realtime. The clip can be in another language than the reply.talker/flow.pt and its ONNX) so that unseen voices come out much closer to the reference. Needs the
latest code: GitHub main or pip install -U "sophon-realtime[trt]" (1.1.0 or later).| Concurrent users (arriving within 10 s) | First audio, median / max | Stutter (median) |
|---|---|---|
| 1 | 0.67 s | ~0 |
| 8 | 0.61 / 0.76 s | 0 |
| 16 | 0.75 / 1.07 s | 0.06 s |
| 24 | 1.36 / 2.11 s | 0.39 s (queueing starts) |
install.sh installs it; with pip, install pip install "sophon-realtime[trt]" (CUDA 13). Without
TensorRT the voice runs in plain PyTorch and gets about 3× slower under load: at 16 users first audio is about 2.4 s with 1–1.5 s of
stutter. Check the server log for Loaded flow-estimator TensorRT engine.[laughter], [breath], [sigh], [cough], <laughter>…</laughter> (laugh while speaking), <strong>…</strong>
(stress a word). Tags are sent to the voice as whole chunks while text streams in, the same way the voice was trained.
Event heard when the tag is written (96 test sentences × 2 runs through the full served pipeline, AST detector; 8–12 sentences per class, so ±1 sentence ≈ ±10 points):
| sigh | breath | cough | laugh | |
|---|---|---|---|---|
| this release | 25% | 50% | 33% | 42% |
| reference tag-tts renderer, same voice model | 42% | 50% | 33% | 50% |
Run one more command next to the engine and GPT-Live clients can connect by changing only the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says "Let me check that", hands the task to your backend and speaks the result in its own words.
cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh # GPT-Live clients: ws://HOST:8780/v1/live/sessions
Details and measurements: live/ in the GitHub repository.
Fine-tune the thinker's LLM backbone with LoRA in ms-swift to change persona, tone, domain knowledge and tool use. Hearing, the voice and turn-taking stay as released, and sound tags such as laughter and coughs are kept. One 80 GB GPU is enough; see finetune/ in the GitHub repository.
| folder | what | licence |
|---|---|---|
thinker/ |
DuplexJev-32B-Para v1.5 with tag writing merged in, FP8 (with the vision tower) | Apache-2.0 |
talker/ |
tag-tts: language model + distilled flow (4 steps, no CFG) + HiFT; based on Fun-CosyVoice3-0.5B | Apache-2.0 |
voices/ |
six reference voices (clip + precomputed prompt) | Apache-2.0 |
[sigh] and <laughter>…</laughter> are still weak; the thinker writes them more often than the voice delivers.thinker/model-v15-00001-of-00016.safetensors 3.9 GBtalker/llm.pt 2.6 GBthinker/model-v15-00005-of-00016.safetensors 2.1 GBthinker/model-v15-00008-of-00016.safetensors 2.1 GBthinker/model-v15-00011-of-00016.safetensors 2.1 GBthinker/model-v15-00014-of-00016.safetensors 2.1 GBthinker/model-v15-00006-of-00016.safetensors 2.1 GBthinker/model-v15-00015-of-00016.safetensors 2.1 GBthinker/model-v15-00009-of-00016.safetensors 2.1 GBthinker/model-v15-00012-of-00016.safetensors 2.1 GBthinker/model-v15-00010-of-00016.safetensors 2.1 GBthinker/model-v15-00013-of-00016.safetensors 2.1 GBthinker/model-v15-00004-of-00016.safetensors 2.1 GBthinker/model-v15-00007-of-00016.safetensors 2.1 GBthinker/model-v15-00003-of-00016.safetensors 2.1 GBthinker/model-v15-00002-of-00016.safetensors 2.0 GBthinker/model-v15-00016-of-00016.safetensors 1.4 GBtalker/flow.pt 1.3 GBtalker/flow.decoder.estimator.fp32.onnx 1.3 GBthinker/visual.safetensors 1.2 GBtalker/CosyVoice-BlankEN/model.safetensors 988 MBtalker/speech_tokenizer_v3.batch.onnx 969 MBtalker/speech_tokenizer_v3.onnx 969 MBthinker/audio/model.safetensors 405 MBtalker/hift.pt 83 MBtalker/campplus.onnx 28 MBvoices/brand_en.pt 138.2 kBvoices/brand_zh.pt 138.2 kBvoices/vivian_zh.pt 125.2 kBvoices/chelsie_zh.pt 102.7 kBvoices/ryan_en.pt 99.4 kBvoices/chelsie_en.pt 77 kB