Wav2Vec2-BART (Base) for English ASR on VoxPopuli - Best WER from Master's Thesis

This repository contains the checkpoint for a SpeechEncoderDecoderModel fine-tuned for Automatic Speech Recognition (ASR) on the English portion of the VoxPopuli dataset. This model achieved the best Word Error Rate (WER) of 8.85% on the VoxPopuli English test set within the experimental framework of the Master's thesis "Effective Training of Neural Networks for Automatic Speech Recognition" by Matej Horník.

The model leverages a pre-trained Wav2Vec2 (Base) encoder facebook/wav2vec2-base-en-voxpopuli-v2 and a pre-trained BART (Base) decoder facebook/bart-base.

Thesis Context

This model is a direct result of work conducted for the Master's thesis:

• Title: Effective Training of Neural Networks for Automatic Speech Recognition
• Author: Matej Horník
• Supervisor: Ing. Alexander Polok
• Institution: Brno University of Technology, Faculty of Information Technology
• Year: 2025
• Thesis Link: Link to thesis PDF

[!NOTE] Link will be available after the thesis defense.

Thesis Abstract (English)

This master's thesis focuses on improving the training efficiency and performance of encoder-decoder transformer models for Automatic Speech Recognition (ASR). It investigates the impact of initialization strategies using pre-trained components (Wav2Vec2, BART), the role of convolutional adapters, and Parameter-Efficient Fine-tuning (PEFT) methods like LoRA and DoRA. Experiments on LibriSpeech and VoxPopuli datasets confirmed that full pre-trained initialization is crucial for best Word Error Rate (WER) and convergence. An optimal number of adapters improved performance, while PEFT (especially LoRA) significantly reduced trainable parameters with comparable accuracy. Domain-specific encoder pre-training proved beneficial, and the encoder-decoder model outperformed a CTC baseline in accuracy, offering practical insights for efficient ASR training.

Model Details

• Encoder: facebook/wav2vec2-base-en-voxpopuli-v2. This is a Wav2Vec2 (Base) model pre-trained by Facebook on 24.1k hours of unlabeled English VoxPopuli data.
• Decoder: facebook/bart-base. This is a standard BART (Base) model.
• Architecture: SpeechEncoderDecoderModel from Hugging Face Transformers.
• Adapters: 3 convolutional adapter layers were added to the encoder's output to better align its temporal resolution with the BART decoder's input requirements.
• Feature Extractor: The Wav2Vec2 feature extractor (initial CNN layers) was kept frozen during fine-tuning, as experiments showed this maintained performance while reducing trainable parameters.

Initial Model Construction

The base model (before fine-tuning for this specific result) was constructed by combining the pre-trained facebook/wav2vec2-base-en-voxpopuli-v2 (encoder) and facebook/bart-base (decoder) using SpeechEncoderDecoderModel.from_encoder_decoder_pretrained. To create the model, code is provided in create_model.py.

python create_model.py

Training Data

Data

The model was fine-tuned on the train split of the English portion of the VoxPopuli dataset (facebook/voxpopuli, config en). Audio data was resampled to 16kHz. Text transcriptions were tokenized using the BART tokenizer and lowercased.

Procedure

The model was fine-tuned using modified run_speech_recognition_seq2seq.py script (provided in the thesis materials, based on Hugging Face's example scripts).

Key Hyperparameters:

• Optimizer: AdamW
• Learning Rate: 1e-4
• LR Scheduler: cosine_with_min_lr (min_lr: 5e-9)
• Warmup Steps: 2000
• Batch Size (per device): 96
• Gradient Accumulation Steps: 1
• Number of Epochs: 20
• Weight Decay: 0.01
• Label Smoothing Factor: 0.05
• Mixed Precision: bf16
• SpecAugment: Applied during training
  • mask_time_prob: 0.25, mask_time_length: 30, mask_time_min_masks: 2
  • mask_feature_prob: 0.3, mask_feature_length: 30, mask_feature_min_masks: 1
• Feature Extractor: Frozen

The full training command can be found in the thesis materials, including the specific arguments used.

Evaluation

The model achieves the following Word Error Rate (WER) on the VoxPopuli English dataset:

For detailed training logs, metrics, and visualizations, please refer to the Weights & Biases report:

How to Use

You can use this model for inference with the Hugging Face transformers library.

from transformers import SpeechEncoderDecoderModel, AutoProcessor
import torch
from datasets import load_dataset

MODEL_ID = "matejhornik/wav2vec2-base_bart-base_voxpopuli-en"
DATASET_ID = "facebook/voxpopuli"
DATASET_CONFIG = "en"
DATASET_SPLIT = "test" # "validation"

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = SpeechEncoderDecoderModel.from_pretrained(MODEL_ID).to(device)

print(f"Using device: {device}\nStreaming one sample from '{DATASET_ID}'"
"(config: '{DATASET_CONFIG}', split: '{DATASET_SPLIT}')...")
streamed_dataset = load_dataset(
    DATASET_ID,
    DATASET_CONFIG,
    split=DATASET_SPLIT,
    streaming=True,
)
sample = next(iter(streamed_dataset))

audio_input = sample["audio"]["array"]
input_sampling_rate = sample["audio"]["sampling_rate"]

inputs = processor(audio_input, sampling_rate=input_sampling_rate, return_tensors="pt", padding=True)
input_features = inputs.input_values.to(device)

with torch.no_grad():
    predicted_ids = model.generate(input_features, max_length=128)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

print(f"\nOriginal: {sample['normalized_text']}")
print(f"Transcribed: {transcription}")

Framework Versions

This model was trained using:

• Python: ^3.10
• Transformers: ~4.46.3
• PyTorch: ~2.5.1
• Datasets: ^3.2.0
• PEFT: ^0.14.0
• Accelerate: ^1.4.0
• Evaluate: ^0.4.3
• WandB: ^0.19.7

Visit the pyproject.toml file for a complete list of dependencies.

Citation

Citation If you use this model or findings from the thesis, please cite:

@mastersthesis{Hornik2025EffectiveTraining,
  author       = {Horník, Matej},
  title        = {Effective Training of Neural Networks for Automatic Speech Recognition},
  school       = {Brno University of Technology, Faculty of Information Technology},
  year         = {2025},
  supervisor   = {Polok, Alexander},
  type         = {Master's Thesis},
  note         = {Online. Available at: \url{https://www.vut.cz/en/students/final-thesis/detail/164401}}
}

Acknowledgements

• My supervisor, Ing. Alexander Polok, for his valuable guidance and support.
• The Hugging Face team for their comprehensive transformers, datasets, and evaluate libraries.
• The creators of Wav2Vec2, BART, and the VoxPopuli dataset.

Contact

For questions, feedback, or collaboration opportunities related to this thesis or any other stuff, feel free to reach out:

• Email: xhorni20@vut.cz / xhorni20@stud.fit.vut.cz
• GitHub: hornikmatej