Phonon 31M (English) — MediaPipe & LiteRT

Self-contained .litertlm package of gradium/phonon-31m-en (ultra-compact 31M-parameter 6-layer Llama FlowLM backbone + 3-layer ConvNext flow-matching head + stateful Mimi neural audio codec decoder, 24 kHz mono PCM output) quantized to INT8 for fast on-device speech synthesis with the MediaPipe Text-to-Speech (TextToSpeech) Task and LiteRT across Android, iOS, macOS, Python, Web, and C++.

At just 84 MB, phonon_int8.litertlm packages the INT8 LiteRT subgraphs (text embedding table, autoregressive FlowLM backbone, flow-matching velocity network, and streaming Mimi codec decoder), the SentencePiece tokenizer, and pre-encoded voice conditioning embeddings into a single memory-mapped file.

Available Model Bundle

You can enumerate all voices packaged inside the loaded .litertlm bundle at runtime using listVoices() / list_voices().

---

MediaPipe Text-to-Speech Task Capabilities

When loaded with the MediaPipe TextToSpeech Task, Phonon 31M supports:

• Ultra-compact single-file .litertlm deployment (84 MB): Produces normalized 24 kHz mono 32-bit floating-point (float32) PCM audio in [-1.0, 1.0] with zero external phonemizer or Python dependencies and a ~120 MB peak runtime memory footprint.
• Low-latency streaming speech synthesis (synthesizeStream / synthesize_stream): Streams AudioChunk buffers as latent frames are generated and decoded by the stateful Mimi codec, using adaptive first-chunk scheduling (1 frame on step 0, 2 frames on steps 1–4, 4 frames thereafter) for ~45 ms Time-To-First-Audio (TTFA).
• Real-time LLM-to-TTS token streaming (openStream / start_stream): Push incremental text deltas (pushText / push_text) directly from an on-device LLM stream and call finishInput / finish_input when done. Speech synthesis begins immediately once the first clause or word-boundary threshold arrives.
• Word-level timestamp alignment (wordTimestamps / word_timestamps): Both unary AudioResult and streaming AudioChunk results include pause-aware WordTimestamp entries (word, startChar, endChar, startSec, endSec) mapped directly to character offsets in the input string for real-time read-along text highlighting.
• Sub-millisecond voice switching: Switch between bundled voice conditioning embeddings per utterance using SynthesisOptions in < 1 ms without reloading model weights.
• Smart clause chunking: Automatically segments multi-sentence paragraphs at clause boundaries while guarding common abbreviations (Dr., Mr., U.S., e.g.) and decimal numbers (3.14).
• Speaking rate control & lock-free barge-in cancellation: Adjust speakingRate ((0.1, 4.0], default 1.0) via pitch-preserving WSOLA time-scale modification, and abort in-flight synthesis within milliseconds via cancel() or native Kotlin Coroutine / Swift Task cancellation.

---

Download

pip install -U huggingface_hub

hf download litert-community/phonon-31m-en phonon_int8.litertlm --local-dir .

---

Usage with MediaPipe Text-to-Speech Task

1. Android (Kotlin)

See the MediaPipe Text-to-Speech Android Guide and Android Sample App for a complete project setup.

Add the Gradle dependency

Add the MediaPipe Audio Tasks dependency to your module's build.gradle.kts (minSdk = 26, arm64-v8a):

dependencies {
    implementation("com.google.mediapipe:tasks-audio:latest.release")
}

Initialize the task and list voices

import com.google.mediapipe.tasks.audio.texttospeech.SynthesisOptions
import com.google.mediapipe.tasks.audio.texttospeech.TextToSpeech
import com.google.mediapipe.tasks.audio.texttospeech.TextToSpeechOptions
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.core.Delegate

val options = TextToSpeechOptions(
    baseOptions = BaseOptions.builder()
        .setModelAssetPath("/data/user/0/com.example.app/files/phonon_int8.litertlm")
        .setDelegate(Delegate.CPU)
        .build(),
    defaultVoiceId = "freya",
    speakingRate = 1.0f,
    cacheDirectory = context.cacheDir.absolutePath,
)

val tts = TextToSpeech.create(options)

// Enumerate bundled voices at runtime:
for (voice in tts.listVoices()) {
    println("${voice.voiceId} (${voice.displayName}) - locale=${voice.locale}, ${voice.sampleRateHz} Hz")
}

Unary synthesis & word-level timestamps

// Non-blocking coroutine synthesis (or use tts.synthesize(...) on a background thread):
val result = tts.synthesizeAsync(
    "Hello from Phonon 31M running on-device with MediaPipe Text to Speech!",
    SynthesisOptions(voiceId = "freya"),
)
// result.samples: FloatArray (24 kHz mono float32 PCM in [-1.0, 1.0])
// result.sampleRateHz: 24000
// result.durationSeconds: Float

// Word-level timestamps for synchronized read-along UI highlighting:
for (wt in result.wordTimestamps) {
    println("Word '${wt.word}' chars=[${wt.startChar}, ${wt.endChar}) time=${wt.startSec}s..${wt.endSec}s")
}

Streaming synthesis & incremental LLM-to-TTS streaming

// 1. Streaming a multi-sentence prompt chunk-by-chunk:
tts.synthesizeStream(
    "Streaming synthesis plays the first audio chunk in ~45 ms. Subsequent chunks arrive while audio plays.",
    SynthesisOptions(voiceId = "freya"),
).collect { chunk ->
    streamingAudioTrack.write(chunk.samples, 0, chunk.samples.size, AudioTrack.WRITE_BLOCKING)
    // chunk.wordTimestamps provides word timing offsets for the current chunk
}

// 2. Incremental token-by-token streaming from an on-device LLM:
tts.openStream(options = SynthesisOptions(voiceId = "freya")).use { stream ->
    stream.pushText("Hello from an ")
    stream.pushText("on-device LLM! ")
    stream.pushText("Audio starts streaming before generation finishes.")
    stream.finishInput()

    while (true) {
        val chunk = stream.readNext() ?: break
        streamingAudioTrack.write(chunk.samples, 0, chunk.samples.size, AudioTrack.WRITE_BLOCKING)
    }
}

// Release native resources when done:
tts.close()

---

2. iOS & macOS (Swift)

See the MediaPipe Text-to-Speech iOS Guide and iOS Sample App.

Add the Swift Package

In Xcode, select File > Add Package Dependencies..., enter https://github.com/google-ai-edge/mediapipe, and add the MediaPipeTasksAudio library to your target.

Initialize and synthesize speech

import Foundation
import MediaPipeTasksAudio

let modelPath = Bundle.main.path(forResource: "phonon_int8", ofType: "litertlm")!
let cacheDir = FileManager.default.temporaryDirectory
    .appendingPathComponent("mediapipe_tts_cache").path

let options = TextToSpeechOptions(
    baseOptions: TTSBaseOptions(
        modelAssetPath: modelPath,
        delegate: .cpu
    ),
    defaultVoiceId: "freya",
    speakingRate: 1.0,
    cacheDirectory: cacheDir
)

let tts = try createTextToSpeech(options: options)

// Enumerate bundled voices:
for voice in try tts.listVoices() {
    print("\(voice.voiceId) - locale=\(voice.locale), \(voice.sampleRateHz) Hz")
}

// 1. Async unary synthesis with word-level timestamps:
let result = try await tts.synthesizeAsync(
    "Hello from Phonon 31M on Apple platforms!",
    options: SynthesisOptions(voiceId: "freya")
)
for wt in result.wordTimestamps {
    print("\(wt.word) [\(wt.startChar)..<\(wt.endChar)]: \(wt.startSec)s - \(wt.endSec)s")
}

// 2. Low-latency streaming synthesis:
for try await chunk in tts.synthesizeStream(
    "Streaming synthesis emits 24 kHz PCM chunks as soon as each latent group decodes.",
    options: SynthesisOptions(voiceId: "freya")
) {
    // Schedule chunk.samples (24 kHz mono Float) onto AVAudioPlayerNode
}

// 3. Incremental LLM-to-TTS streaming:
let stream = try tts.openStream(options: SynthesisOptions(voiceId: "freya"))
defer { stream.close() }

try stream.pushText("Streaming tokens from ")
try stream.pushText("an on-device language model.")
try stream.finishInput()

while let chunk = try stream.readNext() {
    // Play chunk.samples & highlight chunk.wordTimestamps
}

---

3. Python

import soundfile as sf
from mediapipe.tasks.python.audio import text_to_speech
from mediapipe.tasks.python.core import base_options

options = text_to_speech.TextToSpeechOptions(
    base_options=base_options.BaseOptions(
        model_asset_path="phonon_int8.litertlm"
    ),
    default_voice_id="freya",
    speaking_rate=1.0,
)

with text_to_speech.TextToSpeech.create_from_options(options) as tts:
    # List bundled voices
    for voice in tts.list_voices():
        print(f"{voice.voice_id}: locale={voice.locale}, {voice.sample_rate_hz} Hz")

    # 1. Unary synthesis + word-level timestamps
    result = tts.synthesize(
        "Hello from Phonon 31M running on-device with MediaPipe in Python!",
        voice_id="freya",
    )
    sf.write("output.wav", result.samples, result.sample_rate_hz)
    for wt in result.word_timestamps:
        print(f"{wt.word} [{wt.start_char}:{wt.end_char}] {wt.start_sec:.2f}s - {wt.end_sec:.2f}s")

    # 2. Streaming synthesis
    for chunk in tts.synthesize_stream(
        "First sentence streams right away. Second sentence follows seamlessly.",
        voice_id="freya",
    ):
        print(f"Chunk {chunk.chunk_index}: {len(chunk.samples)} samples (final={chunk.is_final})")

    # 3. Incremental LLM token streaming
    with tts.start_stream(voice_id="freya") as stream:
        stream.push_text("Streaming tokens ")
        stream.push_text("directly into Phonon 31M.")
        stream.finish_input()
        for chunk in stream:
            pass  # Play chunk.samples in real time

---

4. Web (JavaScript / TypeScript)

import { FilesetResolver, TextToSpeech } from "@mediapipe/tasks-audio";

const wasmFileset = await FilesetResolver.forAudioTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio@latest/wasm"
);

const tts = await TextToSpeech.createFromOptions(wasmFileset, {
  baseOptions: {
    modelAssetPath: "phonon_int8.litertlm",
    delegate: "CPU",
  },
  defaultVoiceId: "freya",
  speakingRate: 1.0,
});

// List bundled voices
const voices = await tts.listVoices();
console.log("Available voices:", voices);

// 1. Unary synthesis
const result = await tts.synthesize(
  "Hello from Phonon 31M in the browser via MediaPipe WebAssembly!",
  { voiceId: "freya" }
);
// result.samples: Float32Array, result.sampleRateHz: 24000, result.wordTimestamps

// 2. Streaming synthesis
const stream = await tts.synthesizeStream(
  "Streaming speech directly to the Web Audio API.",
  { voiceId: "freya" }
);
for await (const chunk of stream) {
  // Schedule chunk.samples (Float32Array at chunk.sampleRateHz) on AudioContext
}

// 3. Incremental LLM token streaming (finishInput = false)
const llmStream = await tts.synthesizeStream("", { voiceId: "freya" }, /* finishInput= */ false);
await llmStream.pushText("Streaming tokens ");
await llmStream.pushText("from WebGPU LLM into Phonon 31M.");
await llmStream.finishInput();
for await (const chunk of llmStream) {
  // Play chunk.samples and highlight chunk.wordTimestamps
}

tts.close();

---

5. C++

#include "mediapipe/tasks/cc/audio/text_to_speech/text_to_speech.h"

using ::mediapipe::tasks::audio::text_to_speech::SynthesisOptions;
using ::mediapipe::tasks::audio::text_to_speech::TextToSpeech;
using ::mediapipe::tasks::audio::text_to_speech::TextToSpeechOptions;

auto options = std::make_unique<TextToSpeechOptions>();
options->base_options.model_asset_path = "phonon_int8.litertlm";
options->default_voice_id = "freya";
options->speaking_rate = 1.0f;

ABSL_ASSIGN_OR_RETURN(auto tts, TextToSpeech::Create(std::move(options)));

// Unary synthesis:
SynthesisOptions synth_opts{.voice_id = "freya"};
ABSL_ASSIGN_OR_RETURN(auto result, tts->Synthesize("Hello from C++!", synth_opts));
// result.samples (std::vector<float>), result.sample_rate_hz (24000), result.word_timestamps

// Streaming synthesis:
ABSL_ASSIGN_OR_RETURN(auto stream, tts->SynthesizeStream("Streaming from C++.", synth_opts));
while (true) {
  ABSL_ASSIGN_OR_RETURN(auto chunk, stream->ReadNext());
  if (!chunk.has_value()) break;
  // Consume chunk->samples and chunk->word_timestamps
}

---

Configuration Options Reference

TextToSpeechOptions

SynthesisOptions (per-utterance override)

---

License & Attribution

• Original model architecture and weights by Gradium, released under the Apache-2.0 license.
• Converted and quantized for MediaPipe and LiteRT using litert-torch.