w2v-bert-2.0-br
This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.6660
- Wer: 42.4942
- Cer: 13.6525
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 6e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.08
- lr_scheduler_warmup_steps: 500
- training_steps: 8001
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Wer | Cer |
|---|---|---|---|---|---|
| 1.0369 | 0.58 | 500 | 1.2289 | 85.3288 | 32.8021 |
| 0.7211 | 1.16 | 1000 | 0.9727 | 70.1973 | 24.6147 |
| 0.5669 | 1.75 | 1500 | 0.8496 | 64.6176 | 21.7978 |
| 0.4229 | 2.33 | 2000 | 0.7448 | 57.2663 | 19.3988 |
| 0.4352 | 2.91 | 2500 | 0.6749 | 52.9790 | 17.4075 |
| 0.3392 | 3.49 | 3000 | 0.6703 | 50.9678 | 16.8375 |
| 0.2508 | 4.07 | 3500 | 0.6143 | 49.6249 | 16.2547 |
| 0.2303 | 4.65 | 4000 | 0.7121 | 48.4648 | 15.8534 |
| 0.1776 | 5.24 | 4500 | 0.6667 | 47.0777 | 15.2910 |
| 0.1645 | 5.82 | 5000 | 0.6715 | 46.1825 | 14.8910 |
| 0.1304 | 6.4 | 5500 | 0.7212 | 44.2784 | 14.5139 |
| 0.1157 | 6.98 | 6000 | 0.6678 | 44.2721 | 14.3043 |
| 0.0924 | 7.56 | 6500 | 0.6935 | 43.1310 | 13.9171 |
| 0.0517 | 8.14 | 7000 | 0.6746 | 42.8851 | 13.7599 |
| 0.0667 | 8.73 | 7500 | 0.6327 | 42.9733 | 13.8136 |
| 0.0483 | 9.31 | 8000 | 0.6660 | 42.4942 | 13.6525 |
Framework versions
- Transformers 4.39.1
- Pytorch 2.0.1+cu117
- Datasets 2.18.0
- Tokenizers 0.15.2