byt5-small-wikipron-eng-latn-multi-broad-p2g
This model is a fine-tuned version of google/byt5-small on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.1238
- Per: 0.2052
- Gen Len: 8.4891
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 128
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 20.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Per | Gen Len |
|---|---|---|---|---|---|
| 2.0082 | 1.0 | 1177 | 0.4061 | 0.6392 | 8.2917 |
| 0.4295 | 2.0 | 2354 | 0.2953 | 0.5242 | 8.3425 |
| 0.3179 | 3.0 | 3531 | 0.2338 | 0.4552 | 8.4024 |
| 0.255 | 4.0 | 4708 | 0.2011 | 0.4038 | 8.4287 |
| 0.2131 | 5.0 | 5885 | 0.1753 | 0.3669 | 8.4356 |
| 0.1813 | 6.0 | 7062 | 0.1567 | 0.3341 | 8.4336 |
| 0.157 | 7.0 | 8239 | 0.1459 | 0.3098 | 8.4546 |
| 0.1368 | 8.0 | 9416 | 0.1349 | 0.2859 | 8.4531 |
| 0.1202 | 9.0 | 10593 | 0.1302 | 0.2663 | 8.4621 |
| 0.1067 | 10.0 | 11770 | 0.1240 | 0.2514 | 8.4701 |
| 0.0946 | 11.0 | 12947 | 0.1203 | 0.2415 | 8.4734 |
| 0.0857 | 12.0 | 14124 | 0.1180 | 0.2347 | 8.4782 |
| 0.0779 | 13.0 | 15301 | 0.1187 | 0.226 | 8.4827 |
| 0.0709 | 14.0 | 16478 | 0.1180 | 0.2211 | 8.4781 |
| 0.0646 | 15.0 | 17655 | 0.1176 | 0.2147 | 8.4856 |
| 0.0602 | 16.0 | 18832 | 0.1178 | 0.2129 | 8.4858 |
| 0.0563 | 17.0 | 20009 | 0.1200 | 0.2113 | 8.4844 |
| 0.0532 | 18.0 | 21186 | 0.1218 | 0.2069 | 8.4907 |
| 0.0501 | 19.0 | 22363 | 0.1228 | 0.2057 | 8.4891 |
| 0.0486 | 20.0 | 23540 | 0.1238 | 0.2052 | 8.4891 |
Framework versions
- Transformers 4.28.1
- Pytorch 2.0.0+cu117
- Datasets 2.11.1.dev0
- Tokenizers 0.13.2