MSP-VSR

MSP-VSR is the visual speech-recognition model in Multimodal Speech Perception (MSP). It uses an AV-HuBERT encoder and a CTC head to transcribe English speech from silent mouth-region video.

The processor converts frames to grayscale, rescales and normalizes them, resizes to 96 pixels, and applies an 88-pixel crop. MSP-VSR also supplies the visual encoder to MSP-AVSR, where its representations participate in both directions of the bidirectional cross-attention fusion block.

Usage

import torch
from transformers import AutoModelForCTC, AutoProcessor

model_id = "MahmoodAnaam/MSP-VSR"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = processor(videos="sample.mp4", return_tensors="pt")
with torch.inference_mode():
    logits = model(**inputs).logits

text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0]
print(text)

Use footage with a visible, trackable speaking face. For 3-gram beam search, install pyctcdecode and kenlm, then decode with MahmoodAnaam/MSP-Processor-With-LM. Pin Hub revisions when loading custom code in controlled environments.

Training

The selected validation checkpoint recorded loss 1.2109 and WER 61.98%. The recorded stack was Transformers 5.10.2, PyTorch 2.10.0 with ROCm 7.2.4, Datasets 4.0.0, and Tokenizers 0.22.2.

Evaluation

The following tables report MSP-VSR only. Results use MSP-Processor-With-LM; WER is a percentage and lower is better.

LRS2

AVCocktail

Resources

• Source and training configuration
• MSP-VSR demo
• 3-gram LM processor