Whisper-Tiny Portuguese - Full Synthetic Data (Unfiltered)

This model is a fine-tuned version of openai/whisper-tiny for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with all synthetic speech data without quality filtering, representing the maximum data augmentation approach.

Purpose

This model completes the evaluation of synthetic data augmentation strategies for Whisper-Tiny Portuguese by testing whether maximum data volume can compensate for architectural limitations. The results provide important insights:

Key Finding: Unfiltered synthetic data provides better in-domain performance (29.84%) than mid-high filtered (30.11%), but worse than high-quality filtered (29.33%). This reveals a non-monotonic relationship between data volume and performance for Tiny models.

Interestingly, including all data (even low-quality) outperforms the mid-high threshold, suggesting complex interactions between data quality and model capacity.

Model Details

Evaluation Results

This Model (whisper-tiny-cv-full-synthetic-pt)

Comparison with Other Training Configurations (Whisper-Tiny Portuguese)

Key Performance Characteristics

• Second-best in-domain: 29.84% Test WER (only 0.51% worse than best)
• Better than mid-high: Outperforms q ≥ 0.5 threshold despite including low-quality data
• Worse cross-domain than baseline: 46.54% vs 45.83% MLS WER
• Most training steps: 860 max steps (100% more than baseline)
• Largest dataset: 43,834 samples—double the baseline

Complete Portuguese Tiny Model Rankings

Unexpected finding: Unfiltered data ranks #2, outperforming mid-high filtering. This suggests that for Tiny models, either strict filtering (q ≥ 0.8) or no filtering works better than moderate filtering.

Non-Monotonic Quality-Performance Relationship

For Tiny models, the relationship between data quality threshold and performance is non-monotonic:

Performance (CV WER):
  Best    → High-Quality (q≥0.8): 29.33%
  #2      → Unfiltered:          29.84%
  #3      → Mid-High (q≥0.5):    30.11%
  Worst   → CV Only:             30.72%

Interpretation:

• Strict filtering removes problematic samples effectively (high-quality is best)
• No filtering allows the model to see all patterns, including occasional useful low-quality samples
• Moderate filtering (q ≥ 0.5) may remove some useful borderline samples while keeping too much noise

This phenomenon doesn't occur with Large-v3, where mid-high filtering is optimal, suggesting it's specific to limited-capacity architectures.

Training Data

Dataset Composition

WAVe Quality Distribution (For Reference)

Training Procedure

Hyperparameters

Training Infrastructure

• GPU: NVIDIA H200 (140GB VRAM)
• Operating System: Ubuntu 22.04
• Framework: Hugging Face Transformers

Usage

Transcription Pipeline

from transformers import pipeline

transcriber = pipeline(
    "automatic-speech-recognition",
    model="yuriyvnv/whisper-tiny-cv-full-synthetic-pt",
    device="cuda"
)

result = transcriber("path/to/portuguese_audio.wav")
print(result["text"])

Direct Model Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-tiny-cv-full-synthetic-pt")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-tiny-cv-full-synthetic-pt")
model.to("cuda")

audio, sr = librosa.load("path/to/portuguese_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Specifying Language

model.generation_config.language = "pt"
model.generation_config.task = "transcribe"

When to Use This Model

Generally not recommended for production. This model is useful for:

• Research purposes: Understanding non-monotonic quality-performance relationships
• Ablation studies: Complete picture of synthetic data effects on Tiny
• When filtering is unavailable: If you must use unfiltered synthetic data with Tiny

For production use:

• whisper-tiny-high-mixed-pt: Best Tiny (29.33% WER)
• whisper-tiny-cv-only-pt: Simplest setup
• whisper-large-v3-high-mixed-pt: Best overall (7.94% WER)

Research Conclusions

This model completes our analysis of synthetic data augmentation for Portuguese Tiny ASR:

Key Findings:

1. High-quality filtering is optimal for Tiny: Only q ≥ 0.8 provides consistent improvement
2. Unfiltered outperforms mid-high: Non-monotonic relationship specific to small models
3. Cross-domain still degrades: Even best Tiny config (44.18% MLS) trails baseline Large-v3 (22.43%)
4. Model capacity fundamentally limits benefit: Maximum 1.39% improvement regardless of data strategy

Practical Recommendations:

Limitations

• Worse cross-domain than baseline: 46.54% vs 45.83% MLS WER
• Not the best configuration: High-quality filtering is better
• 100% more training steps: Doubled compute for suboptimal results
• Limited architectural capacity: Cannot fully leverage data volume

Citation

This model is part of research on WAVe (Word-Aligned Verification) for synthetic speech quality assessment. While the WAVe methodology paper is currently under review, please cite our previous work that motivated this research:

@article{perezhohin2024enhancing,
  title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
  author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
  journal={IEEE Access},
  year={2024},
  publisher={IEEE}
}

@article{perezhohin2026wave,
  title={WAVe: Word-aligned verification of synthetic speech for ASR},
  author={Perezhohin, Yuriy and Castelli, Mauro},
  journal={Information Sciences},
  pages={123591},
  year={2026},
  publisher={Elsevier}
}

References

• Base Model: openai/whisper-tiny
• Training Data (Real): mozilla-foundation/common_voice_17_0
• Training Data (Synthetic): yuriyvnv/synthetic_transcript_pt
• Whisper Paper: Robust Speech Recognition via Large-Scale Weak Supervision
• Motivating Research: Enhancing ASR with Semantic Audio Filtering (IEEE Access 2024)

License

Apache 2.0