modernisa-v2-byt5-base-lr0.0001
This model is a fine-tuned version of google/byt5-base on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.4744
- Bleu: 30.8745
- Wer: 47.8194
- Cer: 34.4895
- Gen Len: 18.5499
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Bleu | Wer | Cer | Gen Len |
|---|---|---|---|---|---|---|---|
| 0.2696 | 0.09 | 1000 | 0.3027 | 27.8571 | 49.5134 | 34.4149 | 18.5 |
| 0.2518 | 0.17 | 2000 | 0.2857 | 29.2213 | 49.1981 | 34.6336 | 18.5371 |
| 0.2343 | 0.26 | 3000 | 0.2730 | 29.5067 | 49.117 | 34.9795 | 18.5537 |
| 0.2292 | 0.35 | 4000 | 0.2690 | 29.884 | 48.7025 | 34.8015 | 18.5516 |
| 0.2243 | 0.44 | 5000 | 0.2647 | 29.9577 | 48.8466 | 34.7218 | 18.5477 |
| 0.2112 | 0.52 | 6000 | 0.2636 | 30.3115 | 48.3871 | 34.4895 | 18.5477 |
| 0.2118 | 0.61 | 7000 | 0.2555 | 30.6364 | 48.3961 | 34.7455 | 18.5413 |
| 0.205 | 0.7 | 8000 | 0.2508 | 31.0881 | 47.468 | 34.0759 | 18.5269 |
| 0.2049 | 0.78 | 9000 | 0.2471 | 31.1481 | 47.5942 | 34.4133 | 18.5503 |
| 0.2005 | 0.87 | 10000 | 0.2468 | 30.9375 | 47.6392 | 34.281 | 18.5405 |
| 0.1999 | 0.96 | 11000 | 0.2431 | 30.9692 | 47.7023 | 34.4183 | 18.5405 |
| 0.161 | 1.04 | 12000 | 0.2491 | 31.2337 | 47.3238 | 34.1878 | 18.5298 |
| 0.1601 | 1.13 | 13000 | 0.2496 | 31.4422 | 47.3689 | 34.1657 | 18.5371 |
| 0.1606 | 1.22 | 14000 | 0.2459 | 31.4582 | 47.3329 | 34.2386 | 18.5405 |
| 0.1594 | 1.31 | 15000 | 0.2466 | 31.386 | 47.1166 | 34.2912 | 18.5375 |
| 0.1617 | 1.39 | 16000 | 0.2412 | 31.6546 | 46.8373 | 34.0149 | 18.5294 |
| 0.1582 | 1.48 | 17000 | 0.2461 | 31.2924 | 47.4139 | 34.2573 | 18.5503 |
| 0.1572 | 1.57 | 18000 | 0.2425 | 31.1484 | 47.45 | 34.3675 | 18.5499 |
| 0.1565 | 1.65 | 19000 | 0.2424 | 31.6967 | 46.9724 | 34.1047 | 18.5388 |
| 0.1585 | 1.74 | 20000 | 0.2382 | 31.9026 | 47.0175 | 34.281 | 18.558 |
| 0.1522 | 1.83 | 21000 | 0.2365 | 32.1619 | 46.5219 | 33.9369 | 18.5311 |
| 0.156 | 1.92 | 22000 | 0.2381 | 31.7762 | 46.7922 | 33.9572 | 18.5401 |
| 0.1538 | 2.0 | 23000 | 0.2402 | 31.8785 | 46.8012 | 34.2319 | 18.5516 |
| 0.1083 | 2.09 | 24000 | 0.2654 | 31.9905 | 46.603 | 34.0098 | 18.5384 |
| 0.1086 | 2.18 | 25000 | 0.2618 | 31.6257 | 46.9995 | 34.2607 | 18.5409 |
| 0.1092 | 2.26 | 26000 | 0.2658 | 31.4886 | 47.1436 | 34.337 | 18.5422 |
| 0.1086 | 2.35 | 27000 | 0.2666 | 31.8448 | 46.6751 | 34.1217 | 18.5375 |
| 0.1098 | 2.44 | 28000 | 0.2659 | 31.709 | 46.8913 | 34.1946 | 18.5452 |
| 0.1117 | 2.52 | 29000 | 0.2649 | 31.8114 | 46.8913 | 34.1708 | 18.5431 |
| 0.1094 | 2.61 | 30000 | 0.2656 | 31.6955 | 46.8643 | 34.1606 | 18.5375 |
| 0.1077 | 2.7 | 31000 | 0.2637 | 31.5495 | 46.8823 | 34.0064 | 18.5448 |
| 0.1088 | 2.79 | 32000 | 0.2669 | 32.0837 | 46.612 | 33.9504 | 18.5413 |
| 0.1087 | 2.87 | 33000 | 0.2646 | 31.5549 | 47.0806 | 34.2149 | 18.5286 |
| 0.1077 | 2.96 | 34000 | 0.2630 | 32.1129 | 46.4318 | 33.9403 | 18.5452 |
| 0.0652 | 3.05 | 35000 | 0.3360 | 31.3861 | 47.1977 | 34.1149 | 18.5396 |
| 0.0662 | 3.13 | 36000 | 0.3401 | 31.2372 | 47.3869 | 34.203 | 18.552 |
| 0.0666 | 3.22 | 37000 | 0.3389 | 31.3462 | 47.2968 | 34.1759 | 18.5469 |
| 0.0648 | 3.31 | 38000 | 0.3339 | 30.835 | 47.6753 | 34.381 | 18.552 |
| 0.0654 | 3.4 | 39000 | 0.3395 | 31.0958 | 47.7203 | 34.4692 | 18.5524 |
| 0.0663 | 3.48 | 40000 | 0.3318 | 31.126 | 47.5942 | 34.4539 | 18.5499 |
| 0.0648 | 3.57 | 41000 | 0.3397 | 31.0295 | 47.5852 | 34.3539 | 18.5477 |
| 0.0635 | 3.66 | 42000 | 0.3414 | 31.1287 | 47.5491 | 34.4285 | 18.5494 |
| 0.0656 | 3.74 | 43000 | 0.3394 | 30.9225 | 47.6392 | 34.4285 | 18.5563 |
| 0.0625 | 3.83 | 44000 | 0.3420 | 31.2435 | 47.2968 | 34.1674 | 18.5439 |
| 0.0636 | 3.92 | 45000 | 0.3448 | 31.0688 | 47.6843 | 34.3743 | 18.5439 |
| 0.0586 | 4.0 | 46000 | 0.3675 | 31.2353 | 47.441 | 34.2963 | 18.549 |
| 0.0298 | 4.09 | 47000 | 0.4566 | 30.698 | 47.8555 | 34.4319 | 18.5512 |
| 0.0301 | 4.18 | 48000 | 0.4724 | 30.7773 | 47.8374 | 34.3861 | 18.5507 |
| 0.0311 | 4.27 | 49000 | 0.4640 | 31.0878 | 47.6212 | 34.3861 | 18.5503 |
| 0.03 | 4.35 | 50000 | 0.4654 | 30.8319 | 47.8915 | 34.459 | 18.5529 |
| 0.0302 | 4.44 | 51000 | 0.4665 | 30.9236 | 47.9276 | 34.4997 | 18.552 |
| 0.029 | 4.53 | 52000 | 0.4757 | 30.8307 | 47.9456 | 34.4997 | 18.5482 |
| 0.0301 | 4.61 | 53000 | 0.4672 | 30.7983 | 47.9456 | 34.5218 | 18.5473 |
| 0.0294 | 4.7 | 54000 | 0.4715 | 30.8924 | 47.7564 | 34.4353 | 18.5529 |
| 0.0288 | 4.79 | 55000 | 0.4752 | 30.7372 | 47.7924 | 34.4675 | 18.5524 |
| 0.0289 | 4.88 | 56000 | 0.4744 | 30.8554 | 47.8555 | 34.459 | 18.5516 |
| 0.0288 | 4.96 | 57000 | 0.4744 | 30.8745 | 47.8194 | 34.4895 | 18.5499 |
Framework versions
- Transformers 4.30.0.dev0
- Pytorch 1.13.0+cu117
- Datasets 2.12.0
- Tokenizers 0.11.0