Sophon-Realtime-32B-E2E（研究预览）
Sophon-Realtime v1 系列的端到端版：听和说用同一种向量语言
AI降临派（Adventists AI）· Apache-2.0
GitHub · Hugging Face · 魔搭 · English
研究预览。 生产使用请选 Sophon-Realtime-32B-Tag（表现力最强，单卡）或 Sophon-Realtime-32B-Flash（最快、并发最高，单卡）。三个版本是同一个大脑，只有出声方式不同。
这是什么
E2E 版不用单独的语音模型。32B 模型在听你说话时，内部会形成一串连续向量，里面既有说了什么，也有怎么说的（语调、轻重、情绪）。E2E 的说话头直接用这门「向量语言」写出回复，再由解码器还原成声音，所以模型听得懂的语气，它也能说出来。
在 v1 系列里的位置
Sophon-Realtime v1 有三个版本，同一个大脑（VoiceBench 82.8、Big Bench Audio 81.8%，两项开源全双工第一；副语言理解综合第一），区别只在回复怎么变成声音：
| Tag（推荐） | Flash | E2E（本仓库，研究预览） | |
|---|---|---|---|
| NVV-SuperBench（中 / 英） | 0.86 / 0.80 | 0.77 / 0.63 | 0.77 / 0.60 |
| 单路首包 | 约 0.6 秒 | 0.31 秒 | 约 1 秒 |
| 显卡 | 1 张 | 1 张 | 2 张 |
E2E 适合研究端到端的语音表达；生产使用建议选 Tag 或 Flash。系列介绍和完整评测见 v1 系列主页。
本版更新（2026-11）
- 解码器换代，4 步出声。 每块音频的计算量与旧版相同。
- 同一说话头的回复上，与旧版解码器对比：
| 旧版（Sophon-Realtime-v1） | 本版 | |
|---|---|---|
| 长回复错字率（150 条，中 / 英 / 日） | 1.78%（1.03 / 1.32 / 2.99） | 1.51%（0.97 / 0.71 / 2.86） |
| 未见说话人参考音色的相似度 | 0.33 | 0.52 |
| 单路首包（发布包 bench，中位） | 0.57 s | 0.51 s |
- 旧版权重仍保留在 Sophon-Realtime-v1。
部署
两张 80 GB 显卡：思考器一张，说话引擎一张（说话引擎单卡建议 4–5 路并发）。需要 Linux、NVIDIA 驱动 580 及以上。
git clone https://github.com/adventists-ai/sophon-realtime
cd sophon-realtime
bash scripts/install.sh # 建两个虚拟环境，并从魔搭下载本模型
SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E THINKER_GPU=0 TALKER_GPU=1 bash scripts/start.sh
启动后打开 http://<服务器>:8000/ 即可在网页里直接对话，也可以通过 WebSocket 接口（/ws）接入自己的客户端。说明见 GitHub 仓库的 docs/deploy_CN.md。
文件
| 目录 | 内容 |
|---|---|
base/ |
32B 底座（思考器与说话头共用） |
adapters/tag/ |
写回复的 LoRA |
talker/ |
说话头 |
e2/ |
解码器（4 步）与声码器 |
voices/ |
默认音色 |
许可证
代码和权重都是 Apache-2.0。第三方组件保留各自的许可证，见 GitHub 仓库的 NOTICE。
联系
商务合作：jiejin@adventists.ai
Sophon-Realtime-32B-E2E (research preview)
The end-to-end build of the Sophon-Realtime v1 family: hearing and speaking share one vector language. Adventists AI · Apache-2.0
Research preview. For production, use Sophon-Realtime-32B-Tag (most expressive, one GPU) or Sophon-Realtime-32B-Flash (fastest, most users per GPU, one GPU). All three share the same brain and differ only in how the reply becomes sound.
What it is
The E2E build speaks without a separate speech model. While the 32B model listens, it builds a stream of continuous vectors that carries both what was said and how (intonation, emphasis, emotion). The E2E talker writes replies directly in that vector language and a decoder turns it into sound, so the tone the model can hear is the tone it can speak.
Where it fits in the v1 family
Sophon-Realtime v1 has three builds that share one brain (VoiceBench 82.8 and Big Bench Audio 81.8%, #1 among open full-duplex models on both; #1 in paralinguistic understanding). They differ only in how the reply becomes sound:
| Tag (recommended) | Flash | E2E (this repository, research preview) | |
|---|---|---|---|
| NVV-SuperBench (zh / en) | 0.86 / 0.80 | 0.77 / 0.63 | 0.77 / 0.60 |
| First audio, one user | ~0.6 s | 0.31 s | ~1 s |
| GPUs | 1 | 1 | 2 |
E2E is for research on end-to-end expressive speech; for production, use Tag or Flash. Family overview and full benchmarks: v1 family home page.
What's new (Nov 2026)
- New decoder, 4 steps. Same compute per audio chunk as before.
- Against the previous decoder, on replies from the same talker:
| previous (Sophon-Realtime-v1) | this release | |
|---|---|---|
| Long replies, error rate (150 replies, zh / en / ja) | 1.78% (1.03 / 1.32 / 2.99) | 1.51% (0.97 / 0.71 / 2.86) |
| Voice similarity to an unseen reference speaker | 0.33 | 0.52 |
| First audio, one user (release-package bench, median) | 0.57 s | 0.51 s |
- The previous weights stay in Sophon-Realtime-v1.
Deployment
Two 80 GB GPUs: one for the thinker, one for the speaking engine (4–5 concurrent conversations recommended per engine GPU). Linux with NVIDIA driver 580 or newer.
git clone https://github.com/adventists-ai/sophon-realtime
cd sophon-realtime
bash scripts/install.sh # two virtual environments + this model from ModelScope
SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E THINKER_GPU=0 TALKER_GPU=1 bash scripts/start.sh
Then open http://<server>:8000/ to talk in the browser, or connect your own client to the WebSocket API (/ws). See docs/deploy.md.
Files
| folder | contents |
|---|---|
base/ |
32B backbone (shared by the thinker and the talker) |
adapters/tag/ |
reply LoRA |
talker/ |
talker |
e2/ |
decoder (4 steps) and vocoder |
voices/ |
default voice |
License
Code and weights are released under Apache-2.0. Third-party components keep their own licences; see NOTICE in the GitHub repository.
Contact
Business enquiries: jiejin@adventists.ai