distilbert-fr-explorer-mlm
This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:
- Loss: 1.6345
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 65
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.8566 | 1.0 | 611 | 2.5339 |
| 2.5167 | 2.0 | 1222 | 2.3487 |
| 2.3456 | 3.0 | 1833 | 2.2150 |
| 2.2457 | 4.0 | 2444 | 2.1290 |
| 2.1817 | 5.0 | 3055 | 2.0981 |
| 2.1236 | 6.0 | 3666 | 2.0542 |
| 2.0782 | 7.0 | 4277 | 2.0002 |
| 2.0448 | 8.0 | 4888 | 1.9860 |
| 1.9967 | 9.0 | 5499 | 1.9708 |
| 1.9806 | 10.0 | 6110 | 1.9546 |
| 1.9358 | 11.0 | 6721 | 1.9269 |
| 1.9166 | 12.0 | 7332 | 1.8638 |
| 1.8908 | 13.0 | 7943 | 1.8710 |
| 1.8716 | 14.0 | 8554 | 1.8679 |
| 1.8515 | 15.0 | 9165 | 1.8462 |
| 1.8233 | 16.0 | 9776 | 1.8517 |
| 1.813 | 17.0 | 10387 | 1.8010 |
| 1.7974 | 18.0 | 10998 | 1.8096 |
| 1.7827 | 19.0 | 11609 | 1.8032 |
| 1.7693 | 20.0 | 12220 | 1.7867 |
| 1.7494 | 21.0 | 12831 | 1.7882 |
| 1.7417 | 22.0 | 13442 | 1.7830 |
| 1.7252 | 23.0 | 14053 | 1.7942 |
| 1.7041 | 24.0 | 14664 | 1.7697 |
| 1.6983 | 25.0 | 15275 | 1.7566 |
| 1.6817 | 26.0 | 15886 | 1.7391 |
| 1.6803 | 27.0 | 16497 | 1.7389 |
| 1.6689 | 28.0 | 17108 | 1.7165 |
| 1.6558 | 29.0 | 17719 | 1.7475 |
| 1.6523 | 30.0 | 18330 | 1.7136 |
| 1.6387 | 31.0 | 18941 | 1.7033 |
| 1.6352 | 32.0 | 19552 | 1.7056 |
| 1.6221 | 33.0 | 20163 | 1.6910 |
| 1.6051 | 34.0 | 20774 | 1.6908 |
| 1.6083 | 35.0 | 21385 | 1.6688 |
| 1.6011 | 36.0 | 21996 | 1.6876 |
| 1.5903 | 37.0 | 22607 | 1.6900 |
| 1.5896 | 38.0 | 23218 | 1.6942 |
| 1.5772 | 39.0 | 23829 | 1.6957 |
| 1.5731 | 40.0 | 24440 | 1.6805 |
| 1.5727 | 41.0 | 25051 | 1.6717 |
| 1.566 | 42.0 | 25662 | 1.6848 |
| 1.5588 | 43.0 | 26273 | 1.6788 |
| 1.5549 | 44.0 | 26884 | 1.6919 |
| 1.5444 | 45.0 | 27495 | 1.6610 |
| 1.5372 | 46.0 | 28106 | 1.6602 |
| 1.5426 | 47.0 | 28717 | 1.6669 |
| 1.5262 | 48.0 | 29328 | 1.6666 |
| 1.5287 | 49.0 | 29939 | 1.6461 |
| 1.5241 | 50.0 | 30550 | 1.6612 |
| 1.5257 | 51.0 | 31161 | 1.6447 |
| 1.522 | 52.0 | 31772 | 1.6483 |
| 1.5153 | 53.0 | 32383 | 1.6350 |
| 1.5066 | 54.0 | 32994 | 1.6592 |
| 1.5119 | 55.0 | 33605 | 1.6635 |
| 1.5029 | 56.0 | 34216 | 1.6375 |
| 1.5017 | 57.0 | 34827 | 1.6346 |
| 1.5052 | 58.0 | 35438 | 1.6257 |
| 1.4952 | 59.0 | 36049 | 1.6452 |
| 1.5043 | 60.0 | 36660 | 1.6417 |
| 1.5018 | 61.0 | 37271 | 1.6203 |
| 1.492 | 62.0 | 37882 | 1.6358 |
| 1.4955 | 63.0 | 38493 | 1.6450 |
| 1.5017 | 64.0 | 39104 | 1.6373 |
| 1.4972 | 65.0 | 39715 | 1.6345 |
Framework versions
- Transformers 4.28.1
- Pytorch 2.0.0+cu117
- Datasets 2.11.0
- Tokenizers 0.13.2