whisper-base-en-phoneme-ctc

Fine-tuned Whisper model for pronounciation assessment on the speechocean762 dataset. Phone-level pronunciation scoring for English learners. A Whisper base.en encoder with a CTC phone head force-aligns the phones the speaker was meant to say. A gradient-boosted regressor then turns each phone's Goodness of Pronunciation (GOP) features into a score from 0 to 2: 2 correct, 1 heavy accent, 0 wrong or missing.

Phone-level PCC on the speechocean762 test split (47,369 phones): 0.606

Use

hf download vb223/whisper-base-en-phoneme-ctc --local-dir whisper-phoneme
cd whisper-phoneme
pip install -r requirements.txt

from modeling_whisper_phoneme import load

model = load(".")
scores = model.score_utterance(audio, ["HH", "AH0", "L", "OW0"])
# one dict per phone: {"phone", "score", "start", "end"}, times in seconds

audio is 16 kHz mono, up to 30 seconds. Phones are ARPAbet as spelled in speechocean762, where vowels carry a stress digit (mostly 0): "we" is W IY0. config.json lists the full vocabulary.

Reproduce

python modeling_whisper_phoneme.py

Scores every phone in the speechocean762 test split and prints the pooled PCC.

License and attribution

MIT, see LICENSE. This model is built on:

• Whisper base.en by OpenAI, MIT License. The encoder weights in model.safetensors are derived from it, and OpenAI's copyright notice is included in LICENSE.
• speechocean762 by Zhang et al., used for training and evaluation. It is released under CC BY 4.0 on OpenSLR; the Hugging Face copy used here is marked Apache-2.0.

@inproceedings{zhang2021speechocean762,
  title={speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment},
  author={Zhang, Junbo and Zhang, Zhiwen and Wang, Yongqing and Yan, Zhiyong and Song, Qiong and Huang, Yukai and Li, Ke and Povey, Daniel and Wang, Yujun},
  booktitle={Proc. Interspeech 2021},
  year={2021}
}