distilbert_add_pre-training-complete
This model is a fine-tuned version of distilbert-base-uncased on the wikitext wikitext-103-raw-v1 dataset. It achieves the following results on the evaluation set:
- Loss: 5.0239
- Accuracy: 0.2307
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 10
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- training_steps: 300000
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 6.295 | 1.0 | 3573 | 6.0701 | 0.1522 |
| 6.0482 | 2.0 | 7146 | 5.9533 | 0.1565 |
| 5.9799 | 3.0 | 10719 | 5.9008 | 0.1584 |
| 5.9378 | 4.0 | 14292 | 5.8997 | 0.1545 |
| 5.9057 | 5.0 | 17865 | 5.8905 | 0.1536 |
| 5.8811 | 6.0 | 21438 | 5.8646 | 0.1550 |
| 5.8617 | 7.0 | 25011 | 5.8322 | 0.1534 |
| 5.844 | 8.0 | 28584 | 5.8563 | 0.1523 |
| 5.8297 | 9.0 | 32157 | 5.8352 | 0.1548 |
| 5.8175 | 10.0 | 35730 | 5.8136 | 0.1558 |
| 5.8056 | 11.0 | 39303 | 5.8147 | 0.1526 |
| 5.7921 | 12.0 | 42876 | 5.8020 | 0.1548 |
| 5.7777 | 13.0 | 46449 | 5.7891 | 0.1545 |
| 5.7596 | 14.0 | 50022 | 5.7370 | 0.1587 |
| 5.7414 | 15.0 | 53595 | 5.7396 | 0.1604 |
| 5.7243 | 16.0 | 57168 | 5.7490 | 0.1564 |
| 5.6997 | 17.0 | 60741 | 5.7135 | 0.1561 |
| 5.6698 | 18.0 | 64314 | 5.6858 | 0.1620 |
| 5.6398 | 19.0 | 67887 | 5.6735 | 0.1644 |
| 5.6135 | 20.0 | 71460 | 5.6174 | 0.1681 |
| 5.5899 | 21.0 | 75033 | 5.6191 | 0.1684 |
| 5.5699 | 22.0 | 78606 | 5.5977 | 0.1669 |
| 5.5487 | 23.0 | 82179 | 5.6139 | 0.1669 |
| 5.529 | 24.0 | 85752 | 5.5272 | 0.1741 |
| 5.512 | 25.0 | 89325 | 5.5271 | 0.1727 |
| 5.4939 | 26.0 | 92898 | 5.5190 | 0.1721 |
| 5.4765 | 27.0 | 96471 | 5.4824 | 0.1770 |
| 5.4604 | 28.0 | 100044 | 5.5159 | 0.1747 |
| 5.4422 | 29.0 | 103617 | 5.4577 | 0.1807 |
| 5.4243 | 30.0 | 107190 | 5.4546 | 0.1772 |
| 5.408 | 31.0 | 110763 | 5.4297 | 0.1837 |
| 5.3915 | 32.0 | 114336 | 5.4089 | 0.1866 |
| 5.3766 | 33.0 | 117909 | 5.3996 | 0.1848 |
| 5.3594 | 34.0 | 121482 | 5.3974 | 0.1841 |
| 5.3451 | 35.0 | 125055 | 5.3718 | 0.1908 |
| 5.3294 | 36.0 | 128628 | 5.3706 | 0.1878 |
| 5.3155 | 37.0 | 132201 | 5.3677 | 0.1903 |
| 5.2996 | 38.0 | 135774 | 5.2970 | 0.1994 |
| 5.287 | 39.0 | 139347 | 5.3127 | 0.1977 |
| 5.2735 | 40.0 | 142920 | 5.3145 | 0.1955 |
| 5.26 | 41.0 | 146493 | 5.2985 | 0.2017 |
| 5.2487 | 42.0 | 150066 | 5.2661 | 0.2025 |
| 5.2362 | 43.0 | 153639 | 5.2712 | 0.2031 |
| 5.2248 | 44.0 | 157212 | 5.2452 | 0.2049 |
| 5.2115 | 45.0 | 160785 | 5.2325 | 0.2054 |
| 5.1998 | 46.0 | 164358 | 5.2233 | 0.2075 |
| 5.188 | 47.0 | 167931 | 5.1994 | 0.2118 |
| 5.1779 | 48.0 | 171504 | 5.2436 | 0.2069 |
| 5.1664 | 49.0 | 175077 | 5.2203 | 0.2129 |
| 5.1546 | 50.0 | 178650 | 5.1820 | 0.2134 |
| 5.1431 | 51.0 | 182223 | 5.2029 | 0.2122 |
| 5.133 | 52.0 | 185796 | 5.1458 | 0.2140 |
| 5.1226 | 53.0 | 189369 | 5.1757 | 0.2163 |
| 5.1138 | 54.0 | 192942 | 5.1380 | 0.2193 |
| 5.1046 | 55.0 | 196515 | 5.1498 | 0.2178 |
| 5.0984 | 56.0 | 200088 | 5.1094 | 0.2194 |
| 5.0907 | 57.0 | 203661 | 5.1354 | 0.2202 |
| 5.0812 | 58.0 | 207234 | 5.0662 | 0.2256 |
| 5.0748 | 59.0 | 210807 | 5.1163 | 0.2181 |
| 5.067 | 60.0 | 214380 | 5.1193 | 0.2199 |
| 5.0609 | 61.0 | 217953 | 5.0919 | 0.2224 |
| 5.0536 | 62.0 | 221526 | 5.0899 | 0.2239 |
| 5.0491 | 63.0 | 225099 | 5.1125 | 0.2224 |
| 5.0433 | 64.0 | 228672 | 5.0892 | 0.2226 |
| 5.0373 | 65.0 | 232245 | 5.0644 | 0.2260 |
| 5.032 | 66.0 | 235818 | 5.0623 | 0.2253 |
| 5.0283 | 67.0 | 239391 | 5.1004 | 0.2213 |
| 5.0223 | 68.0 | 242964 | 5.0573 | 0.2279 |
| 5.0184 | 69.0 | 246537 | 5.0488 | 0.2271 |
| 5.014 | 70.0 | 250110 | 5.0482 | 0.2280 |
| 5.0102 | 71.0 | 253683 | 5.0600 | 0.2269 |
| 5.0079 | 72.0 | 257256 | 5.0271 | 0.2279 |
| 5.0029 | 73.0 | 260829 | 5.0629 | 0.2267 |
| 4.9994 | 74.0 | 264402 | 5.0304 | 0.2297 |
| 4.9978 | 75.0 | 267975 | 5.0485 | 0.2269 |
| 4.9945 | 76.0 | 271548 | 5.0380 | 0.2306 |
| 4.9917 | 77.0 | 275121 | 5.0590 | 0.2265 |
| 4.9913 | 78.0 | 278694 | 5.0585 | 0.2262 |
| 4.987 | 79.0 | 282267 | 5.0339 | 0.2278 |
| 4.9862 | 80.0 | 285840 | 5.0214 | 0.2305 |
| 4.9841 | 81.0 | 289413 | 5.0393 | 0.2271 |
| 4.983 | 82.0 | 292986 | 5.0200 | 0.2298 |
| 4.9816 | 83.0 | 296559 | 5.0289 | 0.2300 |
| 4.9801 | 83.96 | 300000 | 4.9972 | 0.2332 |
Framework versions
- Transformers 4.26.0
- Pytorch 1.14.0a0+410ce96
- Datasets 2.9.0
- Tokenizers 0.13.2