Whisper-Large-v3 Portuguese - Mid-High Quality Filtered Synthetic Data
This model is a fine-tuned version of openai/whisper-large-v3 for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with WAVe-filtered synthetic speech data using a balanced quality threshold (q ≥ 0.5), including both high-quality and medium-quality samples.
Purpose
This model demonstrates the optimal balance between data quality and quantity for Portuguese ASR. By retaining 87.3% of synthetic samples (high + medium quality), this model achieves:
- 29.3% WER improvement over the CV-only baseline (8.33% vs 11.78%)
- 32.9% better cross-domain generalization on MLS (10.27% vs 15.31%) - best among all configurations
- Best validation loss (0.1040) among all Portuguese Large-v3 variants
- Balanced training efficiency with 805 max steps (87% increase over baseline)
The model is part of a comprehensive study on WAVe (Word-Aligned Verification) filtering for Portuguese ASR, demonstrating that mid-high quality filtering provides the best overall performance, particularly for cross-domain tasks.
Model Details
| Property | Value |
|---|---|
| Base Model | openai/whisper-large-v3 |
| Language | Portuguese (pt) |
| Task | Automatic Speech Recognition (transcribe) |
| Parameters | 1550M |
| Training Data | Common Voice 17.0 + Mid-High Quality Synthetic (q ≥ 0.5) |
| Total Training Samples | 41,047 |
| Sampling Rate | 16kHz |
Evaluation Results
This Model (whisper-large-v3-mixed-pt)
| Metric | Value |
|---|---|
| Validation Loss | 0.1040 |
| Validation WER | 7.73% |
| Test WER (Common Voice) | 8.33% |
| Test WER (MLS) | 10.27% |
| Best Checkpoint | Step 300 |
| Max Training Steps | 805 |
Comparison with Other Training Configurations (Whisper-Large-v3 Portuguese)
| Training Data | Max Steps | Val Loss | Val WER | Test WER (CV) | Test WER (MLS) |
|---|---|---|---|---|---|
| Common Voice Only | 430 | 0.1260 | 11.38% | 11.78% | 15.31% |
| High-Quality (q ≥ 0.8) + CV | 575 | 0.1045 | 7.33% | 7.94% | 12.41% |
| Mid-High (q ≥ 0.5) + CV | 805 | 0.1040 | 7.73% | 8.33% | 10.27% |
| All Synthetic + CV | 860 | 0.1050 | 7.57% | 8.33% | 13.43% |
Key Performance Highlights
- Best cross-domain performance: Lowest MLS WER (10.27%) among all Portuguese configurations
- Best validation loss (0.1040) - optimal model convergence
- Strong in-domain: 8.33% Test WER on Common Voice (29.3% improvement vs baseline)
- Balanced dataset: 87.3% of synthetic data included (19,181 samples)
- Training efficiency: 6% fewer steps than unfiltered while maintaining quality control
Training Data
Dataset Composition
| Source | Samples | Description |
|---|---|---|
| Common Voice 17.0 Portuguese | 21,866 | Real speech from Mozilla's crowdsourced dataset |
| Synthetic Transcript PT (q ≥ 0.5) | 19,181 | WAVe-filtered TTS audio (high + medium quality) |
| Total | 41,047 |
Synthetic Data Generation Pipeline
The synthetic dataset (yuriyvnv/synthetic_transcript_pt) was generated using:
- Transcript Generation: GPT-4o-mini, matching Common Voice word count distribution
- Speech Synthesis: OpenAI TTS-1 model with 9 voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer)
- Quality Filtering: WAVe model with balanced threshold q ≥ 0.5
WAVe Quality Distribution (Portuguese Synthetic Data)
| Quality Level | Samples | Percentage | Used in This Model |
|---|---|---|---|
| High (q ≥ 0.8) | 7,312 | 33.3% | ✓ |
| Medium (0.5 ≤ q < 0.8) | 11,869 | 54.0% | ✓ |
| Low (q < 0.5) | 2,787 | 12.7% | ✗ |
This threshold retains 87.3% of the synthetic dataset (high + medium quality), filtering only the lowest-quality samples while preserving volume for robust cross-domain training.
Training Procedure
Hyperparameters
| Parameter | Value |
|---|---|
| Learning Rate | 5e-6 |
| Batch Size (Global) | 256 |
| Warmup Steps | 200 |
| Max Epochs | 5 |
| Precision | BF16 |
| Optimizer | AdamW (fused) |
| Eval Steps | 50 |
| Metric for Best Model | eval_loss |
Training Infrastructure
- GPU: NVIDIA H200 (140GB VRAM)
- Operating System: Ubuntu 22.04
- Framework: Hugging Face Transformers
Usage
Transcription Pipeline
from transformers import pipeline
transcriber = pipeline(
"automatic-speech-recognition",
model="yuriyvnv/whisper-large-v3-mixed-pt",
device="cuda"
)
result = transcriber("path/to/portuguese_audio.wav")
print(result["text"])
Direct Model Usage
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-mixed-pt")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-mixed-pt")
model.to("cuda")
audio, sr = librosa.load("path/to/portuguese_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)
Specifying Language
model.generation_config.language = "pt"
model.generation_config.task = "transcribe"
Methodology
This model leverages WAVe (Word-Aligned Verification), a word-level quality assessment method for filtering synthetic speech data. Unlike sentence-level filtering approaches, WAVe:
- Aligns each word to its corresponding audio frames using multi-head attention
- Assigns per-word confidence scores via a GLU-based scorer
- Detects localized synthesis errors (mispronunciations, omitted words, prosodic anomalies)
- Achieves 6.5% improvement over sentence-level filtering methods
The balanced threshold (q ≥ 0.5) retains 87.3% of synthetic samples, striking an optimal balance between data volume and quality for robust cross-domain generalization.
When to Use This Model
This model is ideal when:
- Best cross-domain robustness required: Achieves 10.27% MLS WER (best among all Portuguese configurations)
- Balanced performance needed: Strong on both in-domain (8.33%) and cross-domain (10.27%) benchmarks
- Optimal training efficiency: Best validation loss with reasonable compute budget
- Volume + quality: Includes 87.3% of synthetic data while filtering lowest-quality samples
Consider other variants based on your needs:
- whisper-large-v3-high-mixed-pt: Best in-domain (7.94% CV WER), fewer training steps
- whisper-large-v3-cv-fully-synthetic-pt: Maximum data augmentation
Quality vs Quantity Analysis
This model represents the optimal balance point for Whisper-Large-v3 Portuguese:
| Approach | Synthetic Samples | Training Steps | Test WER (CV) | Test WER (MLS) | Best For |
|---|---|---|---|---|---|
| CV Only | 0 | 430 | 11.78% | 15.31% | Speed |
| High-Quality (q≥0.8) | 7,312 | 575 | 7.94% | 12.41% | In-domain |
| Mid-High (q≥0.5) | 19,181 | 805 | 8.33% | 10.27% | Cross-domain |
| Unfiltered | 21,968 | 860 | 8.33% | 13.43% | Volume |
Key insight: The mid-high threshold achieves the best cross-domain generalization (10.27% MLS WER), outperforming even the unfiltered approach by 23.5% while requiring 6% fewer training steps.
Limitations
- Domain specificity: Optimized for general Portuguese; may underperform on technical domains
- Acoustic conditions: Trained on clean speech; noise robustness not guaranteed
- Dialect coverage: Performance may vary across Portuguese regional variants (European vs Brazilian)
Citation
This model is part of research on WAVe (Word-Aligned Verification) for synthetic speech quality assessment. While the WAVe methodology paper is currently under review, please cite our previous work that motivated this research:
@article{perezhohin2024enhancing,
title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
journal={IEEE Access},
year={2024},
publisher={IEEE}
}
@article{perezhohin2026wave,
title={WAVe: Word-aligned verification of synthetic speech for ASR},
author={Perezhohin, Yuriy and Castelli, Mauro},
journal={Information Sciences},
pages={123591},
year={2026},
publisher={Elsevier}
}
References
- Base Model: openai/whisper-large-v3
- Training Data (Real): mozilla-foundation/common_voice_17_0
- Training Data (Synthetic): yuriyvnv/synthetic_transcript_pt
- Whisper Paper: Robust Speech Recognition via Large-Scale Weak Supervision
- Motivating Research: Enhancing ASR with Semantic Audio Filtering (IEEE Access 2024)
License
Apache 2.0