IndiCodec

Audio samples · Code on GitHub

A 1.09 kbps neural speech codec for building speech language models and TTS. It turns speech into two streams of integer tokens and decodes them back to 24 kHz audio.

It was trained mostly on English and 10 Indian languages, and it is strongest on Indian languages: among codecs at about 1 kbps, it keeps the most words in every one of the 10 we tested.

Key features

• Low bitrate: 1.09 kbps, 75 tokens per second.
• Two token streams: a content stream (25 Hz) that carries the words, and an acoustic stream (50 Hz) that carries the voice and detail.
• Split FSQ, not RVQ: each stream is a single FSQ grid. Content is coded first, and one acoustic grid codes what remains, instead of a stack of residual codebooks. Each frame is one id per stream, with no learned codebook.
• 16 kHz in, 24 kHz out: any input sample rate is resampled to 16 kHz.
• One file: all weights are in model.safetensors. Nothing else is downloaded.
• Commercial use allowed: Apache 2.0. All training data is under open licences, none of them non-commercial.

Model details

The content stream quantises w2v-BERT 2.0 features (layer 16). The acoustic stream quantises what a CNN encoder hears beyond the content stream. A transformer decoder with an iSTFT head turns the sum back into audio.

IndiCodec is built on NeuCodec by Neuphonic: it uses NeuCodec's architecture, and training started from its released weights. The two-stream quantiser, the encoder training and the training recipe are new.

Quality

Speech codecs at about 1 kbps, on speakers not seen in training. Every codec is scored on the same clips.

PESQ-WB (higher is better):

Intelligibility and voice (higher is better):

Speaker similarity is the cosine between WavLM-base-plus-sv x-vectors of each decode and its own original.

Words after a round trip (lower is better):

English WER uses Whisper large-v3. Indian-language CER uses the IndicConformer 600M ASR; the original recordings score 5.2 %, so IndiCodec adds 1.0 point. The 10 languages are Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil and Telugu.

Use

hf download prakharkhatri/IndiCodec --local-dir IndiCodec && cd IndiCodec
pip install torch==2.6.0 torchaudio==2.6.0 safetensors soundfile transformers==4.51.* vector-quantize-pytorch==1.17.8 torchtune==0.5.0 torchao==0.9.0 local-attention einops librosa
pip install --no-deps "git+https://github.com/neuphonic/neucodec@ed3e6cd"

import soundfile as sf
from indicodec import IndiCodec

codec = IndiCodec.load("model.safetensors")
wav, sr = sf.read("speech.wav")

content, acoustic = codec.encode(wav, sr)    # token arrays
audio = codec.decode(content, acoustic)      # 24 kHz

Training data

English: Emilia-YODAS (English part only), LibriTTS, LibriTTS-R and VCTK. Indian languages: Rasa, IndicVoices-R and IndicVoices from AI4Bharat. Plus about 15 h of laughs, sighs and coughs from VocalSound. All CC BY 4.0, except VocalSound (CC BY-SA 4.0). No audio is included in this release.

Limits

• Speech only; not for music.
• Not streaming: the encoder and the decoder both attend over the whole clip.

Licence

Apache 2.0. Built on NeuCodec (Apache 2.0). The bundled w2v-BERT 2.0 layers are Meta's, under MIT; its licence text is in indicodec/w2v_bert/LICENSE.