Haru-4B

Code · Technical blog · Technical blog (Japanese)

Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.

• Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
• Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
• Character and voice control. A text card sets the character, a short reference clip sets the voice.
• Multilingual. English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.

Usage

The code is at https://github.com/higurashi-ai/Haru: a PyTorch implementation with a web demo, an MLX implementation with a native macOS app, and a plugin for serving with vLLM-Omni.

With an NVIDIA GPU (about 12 GB of memory):

git clone https://github.com/higurashi-ai/Haru && cd Haru
pip install -e ./haru
python -m haru.server --hf-repo higurashi-ai/Haru-4B

Open http://localhost:8998, pick a character, give the model a voice, and talk. For low latency, we recommend wired headphones and a wired microphone. Bluetooth devices add 100 ms or more.

Bring your own voice. Haru ships without voice clips. Upload or record a clip of the voice the model should speak with: one speaker, 3 seconds to 3 minutes, clean speech. Only use voices you have the right to use.

from haru import DuplexSession, HaruModel
from haru.audio import load_audio, prepare_reference

model = HaruModel.from_pretrained("higurashi-ai/Haru-4B")
voice = prepare_reference(load_audio("voice.wav"))
session = DuplexSession(model, system_prompt=card, reference=voice)

frame = session.start()                  # the model's first 133 ms: frame.audio, frame.text
while not frame.done:
    frame = session.step(user_audio)     # 3200 samples at 24 kHz in, 3200 out

The system prompt is a character card with a fixed layout; the repository's README shows how to build one.

Files

Architecture

Limitations

Haru-4B is a pretrained base model, released for research. It holds a natural conversation, but it is not a reliable source of facts and does not follow instructions the way an assistant does. It can imitate the voice of a reference clip; only use voices you have the right to use.

License

The weights are released under the Apache License 2.0.