E2E-GOP on SpeechOcean762

E2E-GOP assesses English pronunciation at the phoneme level. Given speech audio and the expected phoneme sequence, it returns a continuous score from 0 to 2 for each phoneme, where higher scores indicate better pronunciation.

The WavLM acoustic backbone was fine-tuned for phoneme recognition on LibriSpeech. The pronunciation scorer was then trained on SpeechOcean762. The architecture combines a WavLM encoder and CTC phoneme head with pronunciation features, learned phoneme embeddings, a two-layer bidirectional LSTM, and a regression head.

The repository includes the complete model, processor, and custom model code. It supports inference entirely on CPU; no GPU is required.

Training and evaluation code: E2E-GOP.

Required packages

• PyTorch (torch)
• Transformers (transformers)
• Safetensors (safetensors)
• SoundFile (soundfile)
• NumPy (numpy)
• SciPy (scipy), for audio resampling

Load on CPU

from transformers import AutoModel, AutoProcessor

repo_id = "anonym-coling2027/E2E-GOP-SpeechOcean762"
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    repo_id, trust_remote_code=True
).to("cpu").eval()

Inference

Use a WAV recording and its expected phonemes in stress-free ARPAbet. The example converts stereo audio to mono and resamples it to 16 kHz when needed. This example assumes hello.wav contains the word "hello". Run it after the loading example above.

from math import gcd

import soundfile as sf
import torch
from scipy.signal import resample_poly

audio, sample_rate = sf.read("hello.wav", dtype="float32")
if audio.ndim == 2:
    audio = audio.mean(axis=1)
target_sample_rate = 16000
if sample_rate != target_sample_rate:
    divisor = gcd(sample_rate, target_sample_rate)
    audio = resample_poly(
        audio,
        up=target_sample_rate // divisor,
        down=sample_rate // divisor,
    ).astype("float32")
    sample_rate = target_sample_rate

phonemes = ["HH", "AH", "L", "OW"]
# For multiple words, insert "|" between their phoneme sequences.
token_ids = processor.tokenizer.convert_tokens_to_ids(phonemes)
canonical_token_ids = torch.tensor([token_ids], dtype=torch.long)
token_lengths = torch.tensor([len(token_ids)], dtype=torch.long)
token_mask = torch.tensor(
    [[phoneme != "|" for phoneme in phonemes]], dtype=torch.bool
)

audio_inputs = processor(
    audio,
    sampling_rate=sample_rate,
    return_tensors="pt",
    return_attention_mask=True,
)

with torch.inference_mode():
    output = model(
        **audio_inputs,
        canonical_token_ids=canonical_token_ids,
        token_lengths=token_lengths,
        token_mask=token_mask,
    )

scores = output.logits[0][token_mask[0]].tolist()
scored_phonemes = [phoneme for phoneme in phonemes if phoneme != "|"]
for phoneme, score in zip(scored_phonemes, scores):
    print(f"{phoneme}: {score:.3f}")