hypaai/Hypa-Orpheus-3b-TTS-VC

An open multilingual text-to-speech and zero-shot voice-cloning model for underrepresented languages, built on Orpheus 3B and trained on Hypa-Voices.

Model Description

Hypa-Orpheus 3B (hypaai/Hypa-Orpheus-3b-TTS-VC) is a LoRA-merged 16-bit bfloat16 checkpoint produced by Hypa Intelligence. It is Step 3a of our Hypa-Whispering-Llama roadmap: an independent multilingual voicebox that turns text (and optional reference speech) into 24 kHz waveform output via discrete SNAC audio tokens.

Built by fine-tuning Orpheus 3B on the full Hypa-Voices corpus, the model unifies four speech capabilities as a single next-token prediction problem:

Collection: hypaai/hypa-orpheus

This release documents the two primary artifacts in the collection:

Hypa-Orpheus powers pronunciation in Hypa Dictionary and spoken output in Hypa Translate, supports experimentation in Hypa Labs, and is being integrated into Hypa Keyboard read-aloud. It also serves as the cascaded speech fallback in our longer-term plan to fuse hearing and speech inside Hypa-Whispering-Llama:

User Speech → Hypa-Whisper → Hypa-Llama → Response Text → Hypa-Orpheus → Response Speech

---

Quick Start

Load the merged model (Transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "hypaai/Hypa-Orpheus-3b-TTS-VC"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Orpheus generation requires SNAC token decoding and training-identical control-token prompts. For a complete TTS or voice-cloning pipeline (prompt assembly, SNAC encode/decode, frame deduplication, and WAV output), see the reference handler.py in this repository and the Orpheus TTS project.

vLLM serving

This repository includes a production vLLM inference handler (handler.py) with backward-compatible API support for legacy Hypa Orpheus clients. Serve with:

vllm serve hypaai/Hypa-Orpheus-3b-TTS-VC --dtype bfloat16 --max-model-len 4096

Use bfloat16 at inference to match training numerics.

Load LoRA adapters instead

To continue fine-tuning or stack adapters, use the companion repository:

from peft import PeftModel
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/orpheus-3b-0.1-ft",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs")

See hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs for TensorBoard logs and adapter details.

---

Prompt Format

Hypa-Orpheus treats speech as causal language modeling over interleaved SNAC tokens. Training and inference must share the same prompt structure.

Text prompts

• transcribe (same-language TTS): {speaker}: {text}
• translate (translation-conditioned TTS): {speaker} - {Language}: {text}

In translate rows from Hypa-Voices, speech is in src_lang and text is in tgt_lang. The language tag in the prompt refers to the text side (tgt_lang).

Control tokens

Orpheus wraps content in role and speech markers. Two rules are fixed at train and serve time:

• <BOS> (<|begin_of_text|>) appears once, at the start of the first human turn.
• Every text span ends with <EOT> (<|eot_id|>), including the boundary between two texts in one human turn.

Vanilla TTS sequence:

[start_of_human] <BOS> {speaker}: {text} <EOT> [end_of_human]
[start_of_ai] [start_of_speech] <SNAC codes> [end_of_speech] [end_of_ai]

Voice cloning (continue-speaking layout):

[start_of_human] <BOS> {ref_text} <EOT> {target_text} <EOT> [end_of_human]
[start_of_ai] [start_of_speech] <ref SNAC codes> ...continues into target codes... [end_of_speech] [end_of_ai]

SNAC codes use Orpheus's 7-token-per-frame interleave (one coarse, two mid, four fine codes per frame), each offset into a distinct 4,096-token vocabulary slot above the base Llama vocabulary.

---

Languages Covered

Hypa-Orpheus was trained on 22 languages spanning AfroVoices studio recordings, African-accented English and French, and adapted public speech (including Common Voice-derived English and Pidgin):

Quality varies by language. Core AfroVoices languages received the densest supervision; tail languages (Fongbe, Twi, Urhobo, Ebira, and others) have thinner cross-lingual connectivity and should be validated by native listeners before production use.

---

Training Data

Training data comes from Hypa-Voices, our multilingual audio-text corpus:

Every row pairs speech and text as transcribe (same language) or translate (cross-lingual). Named speaker IDs enable controllable, voice-aware synthesis rather than anonymous audio.

Offline SNAC precompute

The full corpus was normalized to fast-decoding FLAC, then SNAC-encoded offline into integer codes_list shards. Training reads only precomputed tokens, keeping the GPU compute-bound rather than blocked on audio decode or codec forward passes. A curated 8,800-record public subset is released in both FLAC (Hypa-Voices) and SNAC token form (Hypa-Voices-snac).

For corpus statistics (language tiers, speaker concentration, cross-lingual routes, and sequence-length filtering), see:

Hypa-Voices: analysis of the leading multilingual low-resource dataset

---

Training Procedure

The step budget targets roughly one effective pass through the TTS distribution under weighted interleaving (~70% TTS / ~30% VC per drawn batch).

Training used Unsloth with 4-bit QLoRA. Labels use response-only masking: reference SNAC codes in voice-cloning prompts provide context but are not supervised targets.

---

Evaluation and Checkpoint Selection

Interactive training logs:

TensorBoard on Hugging Face

Validation loss improved at every evaluation interval (19 intervals total). Training loss reached its minimum earlier (step 26,604); because checkpoint selection follows held-out performance, the released weights use the validation-best final step 44,334.

Loss and approximate perplexity (exp(loss)) measure uncertainty over the supervised target SNAC stream, not text-only SFT perplexity and not speech naturalness directly. Informal listening checks across held-out speakers and languages agreed with the scalar choice; formal WER/CER, speaker-similarity, and MOS benchmarks are planned for a future evaluation release.

For downstream use, we recommend the merged 16-bit weights in this repository (step 44,334).

---

Intended Use

Direct use cases:

• Multilingual text-to-speech with named speaker control
• Translation-conditioned speech synthesis (text in tgt_lang, speech reflecting src_lang semantics)
• Zero-shot voice cloning from a short reference clip
• Cross-lingual voice cloning when the reference and target languages differ
• Cascaded speech output paired with Hypa-Llama or other text models

Downstream use:

• Continued fine-tuning from the LoRA adapters in the companion repository
• Adapter stacking for domain- or language-specific heads
• Research on codec-token speech modeling and mixed-task TTS/VC training

Out-of-Scope and Limitations

• Not a general chat model. Despite the Llama backbone, this checkpoint is trained for speech-token generation, not open-ended dialogue.
• Prompt sensitivity. Missing <BOS>, duplicate BOS tokens, or omitted internal <EOT> markers put inference off-distribution and can corrupt audio.
• Context ceiling. Training used 2,048 tokens (~25 s of speech). Longer outputs may degrade; voice-cloning rows with two clips hit the limit first.
• Quality varies by language and speaker. Tail languages and speakers with few training hours may produce less stable output.
• No published perceptual benchmark yet. Declining validation loss and informal listening support a stable fine-tune, but intelligibility, naturalness, and cloning fidelity require formal evaluation before strong product claims.
• SNAC dependency. Inference requires the SNAC codec (hubertsiuzdak/snac_24khz) for encode/decode around the language model.

Bias, Risks, and Limitations

This model inherits biases from its base Orpheus weights and from Hypa-Voices, which is weighted toward AfroVoices studio material, English/French pivot translation routes, and adapted public speech. Generated speech should be reviewed for cultural appropriateness in customer-facing applications, especially for underrepresented languages with thinner training coverage.

Do not use this model unsupervised for applications affecting people's rights, health, finances, or wellbeing. Like all generative models, it can produce confident-sounding but incorrect speech.

---

Released Artifacts

• Merged 16-bit model (this repo): hypaai/Hypa-Orpheus-3b-TTS-VC
• LoRA adapters + TensorBoard: hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs
• Base model: unsloth/orpheus-3b-0.1-ft
• Training dataset (public subset): Hypa-Voices collection
• Dataset analysis: Hypa-Voices statistical breakdown
• Upstream projects: Orpheus TTS, SNAC, Unsloth
• Related text model: Hypa-Llama collection

For notebooks, serving code, or contributions, contact chris@hypaintelligence.com or open an issue on Hugging Face.

---

Citation

If you use Hypa-Orpheus 3B or any of the related work, please cite:

@misc{hypaorpheus2026,
  title        = {Hypa-Orpheus 3B: Multilingual Text-to-Speech and Voice Cloning for Underrepresented Languages},
  author       = {{Hypa Intelligence and AfroVoices}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC}},
  note         = {Apache 2.0 License. Collection: \url{https://huggingface.co/collections/hypaai/hypa-orpheus}}
}

If you use the Hypa-Voices training corpus, please also cite:

@misc{hypavoices2026,
  title={Hypa-Voices: Multilingual Low-Resource Audio-Text Dataset},
  author={Hypa Intelligence and AfroVoices},
  year={2026},
  howpublished={Hugging Face collection},
  url={https://huggingface.co/collections/hypaai/hypa-voices}
}

And the SNAC codec:

@inproceedings{siuzdak2024snac,
  title={SNAC: Multi-Scale Neural Audio Codec},
  author={Siuzdak, Hubert and Gr{\"o}tschla, Florian and Lanzend{\"o}rfer, Luca A.},
  booktitle={Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation},
  year={2024}
}

---

License

Released under the Apache License 2.0. See the LICENSE file in this repository for full details.

---

Acknowledgments

• Canopy Labs, for Orpheus TTS and for treating speech generation as language modeling over discrete audio tokens.
• Meta AI / FAIR, for the Llama family and the open-weights ecosystem Orpheus builds on.
• Hubert Siuzdak and collaborators, for SNAC and for open-sourcing the codec.
• Unsloth, for efficient 4-bit QLoRA and Orpheus integration.
• RunPod, for reliable compute during the production fine-tune.
• Mozilla and the Common Voice community, for open speech data used in parts of Hypa-Voices.
• AfroVoices, for the steadfast curation of high-quality recordings across underrepresented languages.
• Hypa Intelligence Research (HaIR), for our commitment to breaking the language barrier.

---

Contact and Contributions

For questions, issues, or contributions, please open an issue in this repository or contact chris@hypaintelligence.com. Contributions are welcome.

---

Closing Remarks

By releasing Hypa-Orpheus, we hope to give researchers and developers practical access to multilingual speech synthesis and voice cloning for underrepresented languages, and to make the data path, training recipe, and checkpoint-selection process transparent from the start.

At Hypa Intelligence, we believe that for AI to be truly aligned with humanity, it must understand and represent all of us, not just a select few. The first step toward this goal is solving the challenge of multilingualism by breaking the language barrier.

Hypa Intelligence remains steadfast in its mission to accelerate the advent of AGI and ASI and to ensure their benefits are globally distributed.

AfroVoices, a subsidiary of Hypa AI, is dedicated to amplifying African voices, languages, and cultures in the intelligence age. Focused on bridging the digital representation gap, AfroVoices curates datasets and resources for African languages, promoting inclusivity and cultural appreciation in AI technologies.

---

Hypa Intelligence • Website • Hugging Face • Updates

Multilingualism is not a feature. It is a prerequisite for AI that represents all of us.