wav2vec2-bert-turkish
This model is a fine-tuned version of facebook/w2v-bert-2.0 on the common_voice_17_0 dataset. It achieves the following results on the evaluation set:
- Loss: 0.3552
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- total_train_batch_size: 8
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.0927 | 0.1724 | 1000 | 0.6278 |
| 0.4967 | 0.3448 | 2000 | 0.5884 |
| 0.3964 | 0.5172 | 3000 | 0.4851 |
| 0.355 | 0.6895 | 4000 | 0.5371 |
| 0.3264 | 0.8619 | 5000 | 0.4579 |
| 0.2979 | 1.0343 | 6000 | 0.4308 |
| 0.2568 | 1.2067 | 7000 | 0.4136 |
| 0.2495 | 1.3791 | 8000 | 0.4711 |
| 0.2422 | 1.5515 | 9000 | 0.4280 |
| 0.2357 | 1.7238 | 10000 | 0.4045 |
| 0.2193 | 1.8962 | 11000 | 0.4194 |
| 0.2087 | 2.0686 | 12000 | 0.4427 |
| 0.1819 | 2.2410 | 13000 | 0.4155 |
| 0.1772 | 2.4134 | 14000 | 0.4012 |
| 0.1739 | 2.5858 | 15000 | 0.3651 |
| 0.172 | 2.7581 | 16000 | 0.4081 |
| 0.1676 | 2.9305 | 17000 | 0.3948 |
| 0.1498 | 3.1029 | 18000 | 0.3587 |
| 0.1299 | 3.2753 | 19000 | 0.4106 |
| 0.1319 | 3.4477 | 20000 | 0.3624 |
| 0.1425 | 3.6201 | 21000 | 0.3551 |
| 0.1362 | 3.7924 | 22000 | 0.3504 |
| 0.1386 | 3.9648 | 23000 | 0.3454 |
| 0.1106 | 4.1372 | 24000 | 0.3632 |
| 0.1069 | 4.3096 | 25000 | 0.3404 |
| 0.1155 | 4.4820 | 26000 | 0.3517 |
| 0.1162 | 4.6544 | 27000 | 0.3315 |
| 0.1121 | 4.8268 | 28000 | 0.3521 |
| 0.1109 | 4.9991 | 29000 | 0.3456 |
| 0.0875 | 5.1715 | 30000 | 0.3507 |
| 0.0963 | 5.3439 | 31000 | 0.3878 |
| 0.0933 | 5.5163 | 32000 | 0.3653 |
| 0.0988 | 5.6887 | 33000 | 0.3427 |
| 0.0912 | 5.8611 | 34000 | 0.3582 |
| 0.0889 | 6.0334 | 35000 | 0.3262 |
| 0.0769 | 6.2058 | 36000 | 0.3548 |
| 0.08 | 6.3782 | 37000 | 0.4327 |
| 0.0821 | 6.5506 | 38000 | 0.3374 |
| 0.0841 | 6.7230 | 39000 | 0.3522 |
| 0.0826 | 6.8954 | 40000 | 0.3499 |
| 0.0773 | 7.0677 | 41000 | 0.3434 |
| 0.07 | 7.2401 | 42000 | 0.3453 |
| 0.0695 | 7.4125 | 43000 | 0.3455 |
| 0.073 | 7.5849 | 44000 | 0.3614 |
| 0.0705 | 7.7573 | 45000 | 0.3209 |
| 0.0759 | 7.9297 | 46000 | 0.3455 |
| 0.0599 | 8.1021 | 47000 | 0.3237 |
| 0.0617 | 8.2744 | 48000 | 0.3298 |
| 0.0605 | 8.4468 | 49000 | 0.3684 |
| 0.0594 | 8.6192 | 50000 | 0.3623 |
| 0.0631 | 8.7916 | 51000 | 0.3582 |
| 0.0625 | 8.9640 | 52000 | 0.3469 |
| 0.0504 | 9.1364 | 53000 | 0.3462 |
| 0.0502 | 9.3087 | 54000 | 0.3417 |
| 0.0551 | 9.4811 | 55000 | 0.3526 |
| 0.0548 | 9.6535 | 56000 | 0.3359 |
| 0.0563 | 9.8259 | 57000 | 0.3581 |
| 0.056 | 9.9983 | 58000 | 0.3421 |
| 0.042 | 10.1707 | 59000 | 0.3349 |
| 0.05 | 10.3430 | 60000 | 0.3552 |
Framework versions
- Transformers 4.44.0
- Pytorch 2.4.0+cu121
- Datasets 2.18.0
- Tokenizers 0.19.1