Whisper turbo for verbatim oral reading (v0, experimental)

A fine-tune of openai/whisper-large-v3-turbo that transcribes a child reading a passage aloud as it was said, not as it should have been said. It is part of Fluente, a one-minute oral reading fluency check that scores words correct per minute for Portuguese-speaking classrooms.

General ASR models are trained to produce clean text. For reading assessment that is a bug: a model that "fixes" a misread word, drops a hesitation or swallows a false start hides exactly the errors a teacher needs to see. This model was trained to keep them.

Results

All engines were scored with the same deterministic alignment on the Fluente benchmark. The benchmark has 125 scripted readings of 25 passages (pt-BR and en-US), with a known answer key. Neither the voice nor the passages were seen in training. Precision and recall refer to the reading errors the app flags, compared with the answer key.

What this shows:

• Fine-tuning made the model more verbatim. It keeps twice as many hesitations, keeps more false starts, and turns fewer made-up words into real ones.
• There is a trade-off. The model also flags more errors that did not happen, so precision drops from 91% to 84%. The base turbo model is the better choice for word-count accuracy; this model is the better choice for catching what the child actually said.
• Open models are competitive. The base turbo model is roughly at parity with a commercial API under the same alignment.

Training

• Data: 4,224 synthetic readings (≈9.8 h) of 90 original passages, written for this purpose and disjoint from the benchmark. The passages are 60 pt-BR and 30 en-US, for grades 1–5. About two-thirds of the readings contain injected reading errors: misread pseudo-words, substituted words, skips, repetitions, false starts with self-correction, and hesitations. The labels are the verbatim spoken text. The audio was synthesized with multiple neural TTS voices, with randomized rate and pitch.
• Setup: encoder frozen, decoder trained for 1 epoch. Learning rate 1e-5, effective batch 16, fp16, on one RTX 2060 Super (8 GB), in 18 min.
• Inference: the long audio is split at the longest pauses into segments of ≤25 s before transcription. The fixed 30 s chunking produced duplicated text at chunk boundaries with this model.

Limitations: read before using

• No real children yet. Both training and evaluation use synthetic adult-sounding voices. Results on real children's speech are unknown and are likely worse. This is a v0 that tests the method, not a validated assessment tool.
• The model outputs no timestamp tokens. Word timings come from cross-attention (return_timestamps="word").
• The model must not be used to make high-stakes decisions about a child without a teacher reviewing the result.

Next step

The next version will train on readings reviewed by teachers: every confirmation or correction a teacher makes becomes a word-level label. Those readings will be collected with family consent, under Brazil's LGPD art. 14. The metric we will track is a learning curve: agreement with teachers, and the share of readings that need no review, as a function of the number of teacher-reviewed readings.

Usage

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="NicoDeVargas/whisper-turbo-oral-reading-v0", device=0)
asr("reading.wav", return_timestamps="word", generate_kwargs={"language": "portuguese", "task": "transcribe"})