Whisper Small Pt-Br - RFard

This model is a fine-tuned version of openai/whisper-small on the Common Voice 17.0 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2593
• Wer: 17.3791

Model description

Whisper Small Pt-Br - RFard is an automatic speech recognition (ASR) model based on Whisper Small by OpenAI. It has been fine-tuned on the Common Voice 17.0 dataset from Mozilla for the Portuguese language (pt-BR), making it more efficient at transcribing speech to text in this language.

The model uses a transformer-based encoder-decoder architecture optimized for ASR tasks, leveraging Whisper's structure to improve transcription accuracy across various audio sources, including different accents and regional variations of Brazilian Portuguese.

With a Word Error Rate (WER) of 17.38%, the model performs well in transcription tasks but may struggle with noisy audio or overlapping speech.

Intended uses & limitations

This model is designed for automatic speech recognition (ASR) in Brazilian Portuguese, making it suitable for tasks such as speech-to-text transcription, voice assistants, automatic subtitles, and other applications that require converting spoken language into written text.

Limitations

The main limitation of this model is that it was not trained for more epochs due to hardware constraints. Extending the training process could further improve its accuracy and robustness, especially in challenging audio conditions such as noisy environments or overlapping speech.

If you are interested in collaborating on further development and improving the model’s performance, feel free to reach out—I am open to cooperation!

Training and evaluation data

For the experiments, the Common Voice 17.0 dataset was used, which was loaded and segmented into two subsets: the training set, composed of 31,432 samples, and the test set, with 9,467 samples.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 2
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.50.0
• Pytorch 2.6.0+cu124
• Datasets 3.4.1
• Tokenizers 0.21.1

Contact

For any inquiries, collaborations, or contributions to the model’s development, feel free to reach out:

📧 Email: rodrigo.correa.fardin@gmail.com

I am open to discussions and potential improvements to the model! 🚀