Whisper-Large-v3 Portuguese - CAPES Baseline (State-of-the-Art Comparison)

This model is a fine-tuned version of openai/whisper-large-v3 for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with the CAPES filtered synthetic dataset using the original filtering methodology from our previous IEEE Access 2024 paper.

Purpose

This model serves as a direct comparison baseline with the current state-of-the-art Portuguese Whisper model (my-north-ai/whisper-large-v3-pt), which was trained using the same CAPES dataset and filtering approach. The purpose is to:

1. Replicate state-of-the-art performance: Uses the same CAPES synthetic dataset (derived from academic thesis recordings) and filtering methodology as the my-north-ai model
2. Establish comparison baseline: Provides a controlled starting point to evaluate the effectiveness of the new WAVe filtering methodology
3. Benchmark against established methods: Demonstrates performance using sentence-level filtering (the previous state-of-the-art approach)

What is CAPES?

CAPES is a synthetic Portuguese speech dataset generated from academic thesis recordings using our previous methodology (Perezhohin et al., IEEE Access 2024). Key characteristics:

• Source: Academic thesis transcripts (longer, more complex utterances than Common Voice)
• Original size: ~55k synthetic samples
• Filtering: Sentence-level quality assessment (our previous approach)
• Filtered size: 33.2k samples retained after filtering
• Usage: Combined with Common Voice 17.0 for ASR training

This dataset and filtering methodology was used to create the my-north-ai/whisper-large-v3-pt model, which achieved state-of-the-art results for Portuguese Whisper at the time.

Model Details

Evaluation Results

This Model (whisper-large-v3-cv-capes-fs024-IEEE-pt)

Comparison with WAVe Filtering and Other Approaches

Key Performance Characteristics

• State-of-the-art replication: Achieves comparable performance to my-north-ai/whisper-large-v3-pt
• Strong in-domain: 8.43% Test WER on Common Voice
• Baseline cross-domain: 13.54% MLS WER (establishes comparison point)
• Large dataset: Uses 55k total samples (largest among comparisons)
• Most training steps: Requires 1,080 max steps

Comparison with my-north-ai/whisper-large-v3-pt

This model replicates the training methodology used in my-north-ai/whisper-large-v3-pt:

Both models use the same fundamental approach, making this an ideal baseline to evaluate the improvements from WAVe word-level filtering.

Training Data

Dataset Composition

CAPES Dataset Characteristics

The CAPES synthetic dataset differs from our GPT-4o-generated data:

• Source transcripts: Academic thesis recordings (longer, more complex utterances)
• Generation: Previous TTS methodology
• Filtering: Sentence-level quality assessment (not word-level)
• Quality: Contains synthesis errors that may hide within extended passages

Training Procedure

Hyperparameters

Training Infrastructure

• GPU: NVIDIA H200 (140GB VRAM)
• Operating System: Ubuntu 22.04
• Framework: Hugging Face Transformers

Usage

Transcription Pipeline

from transformers import pipeline

transcriber = pipeline(
    "automatic-speech-recognition",
    model="yuriyvnv/whisper-large-v3-cv-capes-fs024-IEEE-pt",
    device="cuda"
)

result = transcriber("path/to/portuguese_audio.wav")
print(result["text"])

Direct Model Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-cv-capes-fs024-IEEE-pt")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-cv-capes-fs024-IEEE-pt")
model.to("cuda")

audio, sr = librosa.load("path/to/portuguese_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Specifying Language

model.generation_config.language = "pt"
model.generation_config.task = "transcribe"

When to Use This Model

This model is ideal when:

• Replicating state-of-the-art: Want to use the established CAPES methodology
• Comparison baseline: Evaluating new filtering approaches against established methods
• Maximum dataset size: Prefer larger synthetic datasets (55k total samples)

Consider WAVe-filtered alternatives for better performance:

• whisper-large-v3-cv-capes-filtered-pt: 49% better cross-domain (6.89% vs 13.54% MLS), 18% fewer steps
• whisper-large-v3-mixed-pt: Best overall with newer synthetic data

Why WAVe Filtering Improves on This Approach

The successor model whisper-large-v3-cv-capes-filtered-pt applies WAVe word-level filtering to the same CAPES dataset and achieves:

• 5.7% better in-domain WER (7.95% vs 8.43%)
• 49% better cross-domain WER (6.89% vs 13.54%) - dramatic improvement
• 18% fewer training steps (880 vs 1,080)
• 30% less data (23k vs 33k synthetic samples)

This dramatic improvement demonstrates that word-level filtering is more effective than sentence-level filtering for detecting synthesis errors in longer utterances.

Limitations

• Sentence-level filtering: Cannot detect localized synthesis errors within long utterances
• Suboptimal cross-domain: 13.54% MLS WER vs 6.89% with WAVe filtering
• Training inefficiency: Requires 1,080 steps vs 880 with WAVe filtering
• Data volume: Uses more data but achieves worse performance than filtered approaches

Citation

This model uses the CAPES dataset and filtering methodology from our previous work:

@article{perezhohin2024enhancing,
  title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
  author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
  journal={IEEE Access},
  year={2024},
  publisher={IEEE}
}

References

• Base Model: openai/whisper-large-v3
• Training Data (Real): mozilla-foundation/common_voice_17_0
• Comparison Model: my-north-ai/whisper-large-v3-pt
• Improved Version: whisper-large-v3-cv-capes-filtered-pt (with WAVe filtering)
• Whisper Paper: Robust Speech Recognition via Large-Scale Weak Supervision
• CAPES Methodology: Enhancing ASR with Semantic Audio Filtering (IEEE Access 2024)

License

Apache 2.0