MSP-ASR

MSP-ASR is the audio model in Multimodal Speech Perception (MSP). It fine-tunes a Wav2Vec2 encoder with a CTC head for English speech recognition and supports greedy decoding or the companion 3-gram language model.

MSP-ASR also supplies the audio encoder to MSP-AVSR. There, its representations participate in both directions of the bidirectional cross-attention fusion block; the standalone checkpoint remains audio-only.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "MahmoodAnaam/MSP-ASR"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = processor(audio="sample.wav", return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits

text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0]
print(text)

For beam search, install pyctcdecode and kenlm, then load MahmoodAnaam/MSP-Processor-With-LM and call batch_decode(logits.cpu().numpy()). Pin Hub revisions when loading custom code in controlled environments.

Training

The selected validation checkpoint recorded loss 0.3481 and WER 20.40%. The recorded stack was Transformers 5.10.2, PyTorch 2.10.0 with ROCm 7.2.4, Datasets 4.0.0, and Tokenizers 0.22.2.

Evaluation

The following tables report MSP-ASR only. Results use MSP-Processor-With-LM; WER is a percentage and lower is better.

LRS2

AVCocktail

Resources

• Source and training configuration
• MSP-ASR demo
• 3-gram LM processor