RubatoCodec

RubatoCodec is a neural speech codec that divides speech into a fixed number of variable-duration events. Each event carries discrete code indices and an explicit duration. Event placement adapts to speech content, while the number of events is determined exactly by the utterance length, so the token sequence length a speech generator must produce is predictable. A causal decoder renders completed events incrementally, which makes the codec suitable for streaming text-to-speech.

The name comes from the musical term rubato: local timing flexes while the overall tempo is preserved.

• Adaptive Firing with an exact event budget. Multi-scale latent dissimilarity and waveform spectral flux guide cumulative allocation on a 50-Hz latent grid. For T frames the event count is exactly max(1, min(T, round(0.25 · T))), giving 12.5 events/s. No learned boundary predictor, boundary labels, or rate loss is used.
• Streaming decoding. Events can be decoded one at a time or in chunks. Event-by-event decoding takes about 11 ms per event on an NVIDIA H200, well below the ~80 ms of audio each event represents on average.
• Hybrid residual quantizer with depth control. A semantically supervised FSQ first layer is followed by 15 DAC-style residual VQ layers, each with 4,096 entries. Any prefix depth from 1 to 16 can be decoded, giving code-index bitrates of 0.15–2.4 kbit/s plus duration information.
• Multilingual. Trained on approximately 375,000 hours of speech, mostly English, Japanese, and Chinese, with 16-kHz input and 44.1-kHz output.

Model details

Usage

Install the package from the GitHub repository:

git clone https://github.com/third-intelligence/RubatoCodec
cd RubatoCodec
uv sync

Encode and decode:

import soundfile as sf
import torch

from rubatocodec import RubatoCodec

codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda")

data, sr = sf.read("speech.wav", dtype="float32")  # mono waveform
wav = torch.from_numpy(data)

enc = codec.encode(wav, sr=sr)       # {"codes": (16, K) int64, "durs": (K,) int64}
audio = codec.decode(enc)            # (1, 1, T_audio) at 44.1 kHz, full depth
audio = codec.decode(enc, depth=8)   # lower bitrate with the first 8 layers

sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr)

durs gives each event's duration in 20-ms grid frames. Loading also downloads facebook/w2v-bert-2.0, which is used only for encoding.

Streaming decoding, inference settings, the code format, and training are documented in the repository: inference guide, architecture, training.

Evaluation

RubatoCodec is evaluated on LibriSpeech test-clean, VCTK, JVS nonpara30, JVNV, and seed-tts-eval test-zh against 13 other codecs. At comparable bitrates, it shows the best intelligibility preservation among the compared codecs on the English and Japanese benchmarks and high predicted quality, while the strongest comparators achieve higher PESQ-WB and STOI.

dWER/dCER compare ASR transcripts of the reconstructed and original audio (Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is cosine similarity of speaker embeddings from a fine-tuned WavLM-Large speaker verification model. Bitrates include duration information.

The tables below show a subset of the compared codecs. Bold and underline mark the best and second-best scores within each bitrate band.

LibriSpeech test-clean (English)

JVNV (Japanese emotional speech with nonverbal vocalizations)

On expressive speech, RubatoCodec keeps dCER below 2% at both bitrates, while every other compared codec exceeds 3%.

Quality versus bitrate (RubatoCodec is marked with a red star)

Adaptive event placement. Each rectangle in the lower strip is one event. Longer events cover pauses and low-energy regions, while speech receives shorter events, with the same 12.5 events/s budget in every utterance.

The full protocol, all compared codecs and operating points, and throughput measurements are in the evaluation document and the paper.

Training data

Approximately 375,000 hours of public and internal speech. Public sources include Emilia-YODAS, HiFiTTS-2 (44-kHz subset), ReazonSpeech, The People's Speech (excluding ShareAlike-licensed data), VoxPopuli, Multilingual LibriSpeech (excluding English), Omnilingual ASR Corpus, and IndicVoices. English, Japanese, and Chinese account for approximately 44%, 27%, and 13% of the data, and other languages for the remaining 16%.

Limitations

• RubatoCodec is trained on speech only and is intended for speech. It is not designed for music or general audio.
• Encoding requires the whole utterance, because the event budget is defined at the utterance level. Only decoding is streaming.
• Signal-fidelity metrics (PESQ-WB, STOI) are below those of the strongest compared codecs at similar bitrates.
• Evaluation covers English, Japanese, and Mandarin, all of which appear in the training data. Performance on other languages has not been evaluated.
• Like other speech codecs, RubatoCodec can be used in speech generation systems capable of voice cloning. Do not use it to impersonate people or to create deceptive content without consent.

Citation

@misc{arata2026rubatocodecbudgetedadaptivespeech,
      title={RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding},
      author={Chihiro Arata and Shohei Taniguchi and Gouki Minegishi and Kan Hatakeyama-Sato},
      year={2026},
      eprint={2610.29434},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2610.29434},
}

Acknowledgements

RubatoCodec builds on X-Codec-2.0 and uses w2v-BERT 2.0 as its frozen semantic encoder. It also adapts components from BigCodec, DAC, Inworld TTS, and BigVGAN. See the repository for the full list and their licenses.

License

The model weights and code are released under the Apache 2.0 License. Third-party components retain their respective licenses.