Fluent Books: sound-check models

The models behind "Say it" and Speak in the Fluent Books app: a small student that hears which sounds were said (CTC over phonemes) and heads that tell a wrong sound from a right one. They run on the device (ONNX Runtime); no audio leaves it.

Each file is an archive in the FBMA1 format of the app (tools/alignment/pack.py in the project): a header with the names and sizes, then the files. The app checks the archive's size and SHA-256, then every file inside, before it uses a set. f1 is the default; f1b (British speech) and stream (live looks) are offered in test builds.

What they are

• f1: DistilHuBERT with four layers taught to read phonemes from the wav2vec2 phoneme model facebook/wav2vec2-lv-60-espeak-cv-ft; its heads (heads_error.onnx, heads_stress.onnx) flag a wrong th, w, ng, i/ɪ, æ and a stressed syllable that was not stressed.
• f1b: F1 continued on British speech; an adapter gives F1's heads the hidden state they expect.
• stream: a causal student for looks while the reader is still speaking; the same heads.

Training data and licences

DistilHuBERT (NTU SPML, Apache-2.0); facebook/wav2vec2-lv-60-espeak-cv-ft (Meta AI, Apache-2.0) as the teacher; Kokoro-82M (hexgrad, Apache-2.0) renders; LibriSpeech (CC BY 4.0), speechocean762 (CC BY 4.0), Common Voice 26.0 (CC0-1.0), RIRS_NOISES (Apache-2.0); VCTK (CC BY 4.0) for checks only. Each archive has a NOTICE.txt with the attributions and the licence text. The licence of the weights is stated as Apache-2.0 on the owner's reading of these sources; check it against yours before relying on it.

Limits

Not measured on Russian speakers. The word-quality head and the omission tables of the app are calibrated for F1; with F1B and the streaming student they are lent, not validated. The sets are the sound-check models of the Fluent Books app (iOS and macOS); they are not a general speech recogniser.