mobilebert_add_pre-training-complete
This model is a fine-tuned version of google/mobilebert-uncased on the wikitext wikitext-103-raw-v1 dataset. It achieves the following results on the evaluation set:
- Loss: 2.9849
- Accuracy: 0.4607
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 10
- distributed_type: multi-GPU
- num_devices: 2
- total_train_batch_size: 128
- total_eval_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- training_steps: 300000
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 4.8119 | 1.0 | 1787 | 4.3700 | 0.3199 |
| 4.2649 | 2.0 | 3574 | 4.0930 | 0.3445 |
| 4.0457 | 3.0 | 5361 | 3.9375 | 0.3545 |
| 3.9099 | 4.0 | 7148 | 3.8534 | 0.3644 |
| 3.8193 | 5.0 | 8935 | 3.7993 | 0.3669 |
| 3.7517 | 6.0 | 10722 | 3.7414 | 0.3730 |
| 3.6983 | 7.0 | 12509 | 3.6737 | 0.3818 |
| 3.6565 | 8.0 | 14296 | 3.6657 | 0.3794 |
| 3.619 | 9.0 | 16083 | 3.6129 | 0.3869 |
| 3.5899 | 10.0 | 17870 | 3.5804 | 0.3910 |
| 3.5597 | 11.0 | 19657 | 3.5432 | 0.3964 |
| 3.5329 | 12.0 | 21444 | 3.5397 | 0.3958 |
| 3.5088 | 13.0 | 23231 | 3.4896 | 0.4011 |
| 3.4904 | 14.0 | 25018 | 3.4731 | 0.4000 |
| 3.4703 | 15.0 | 26805 | 3.4971 | 0.3994 |
| 3.4533 | 16.0 | 28592 | 3.4609 | 0.4049 |
| 3.4369 | 17.0 | 30379 | 3.4411 | 0.4067 |
| 3.423 | 18.0 | 32166 | 3.4219 | 0.4066 |
| 3.4084 | 19.0 | 33953 | 3.4477 | 0.4014 |
| 3.3949 | 20.0 | 35740 | 3.4013 | 0.4087 |
| 3.3811 | 21.0 | 37527 | 3.3642 | 0.4130 |
| 3.3688 | 22.0 | 39314 | 3.4173 | 0.4031 |
| 3.3598 | 23.0 | 41101 | 3.4018 | 0.4101 |
| 3.3484 | 24.0 | 42888 | 3.3499 | 0.4143 |
| 3.3363 | 25.0 | 44675 | 3.3675 | 0.4119 |
| 3.3274 | 26.0 | 46462 | 3.3562 | 0.4154 |
| 3.3161 | 27.0 | 48249 | 3.3487 | 0.4159 |
| 3.3073 | 28.0 | 50036 | 3.3293 | 0.4159 |
| 3.2991 | 29.0 | 51823 | 3.3317 | 0.4160 |
| 3.2899 | 30.0 | 53610 | 3.3058 | 0.4183 |
| 3.2814 | 31.0 | 55397 | 3.2795 | 0.4235 |
| 3.2734 | 32.0 | 57184 | 3.3185 | 0.4143 |
| 3.266 | 33.0 | 58971 | 3.2682 | 0.4268 |
| 3.2578 | 34.0 | 60758 | 3.3145 | 0.4181 |
| 3.2506 | 35.0 | 62545 | 3.2726 | 0.4230 |
| 3.2423 | 36.0 | 64332 | 3.2735 | 0.4218 |
| 3.2359 | 37.0 | 66119 | 3.2845 | 0.4175 |
| 3.2293 | 38.0 | 67906 | 3.3067 | 0.4193 |
| 3.2207 | 39.0 | 69693 | 3.2586 | 0.4257 |
| 3.2138 | 40.0 | 71480 | 3.2543 | 0.4250 |
| 3.2077 | 41.0 | 73267 | 3.2395 | 0.4226 |
| 3.202 | 42.0 | 75054 | 3.2224 | 0.4270 |
| 3.1964 | 43.0 | 76841 | 3.2562 | 0.4234 |
| 3.1925 | 44.0 | 78628 | 3.2544 | 0.4251 |
| 3.1865 | 45.0 | 80415 | 3.2043 | 0.4353 |
| 3.1812 | 46.0 | 82202 | 3.2280 | 0.4286 |
| 3.1744 | 47.0 | 83989 | 3.2174 | 0.4276 |
| 3.1699 | 48.0 | 85776 | 3.1972 | 0.4317 |
| 3.1652 | 49.0 | 87563 | 3.2016 | 0.4302 |
| 3.1609 | 50.0 | 89350 | 3.2018 | 0.4338 |
| 3.1548 | 51.0 | 91137 | 3.1950 | 0.4327 |
| 3.1508 | 52.0 | 92924 | 3.2128 | 0.4279 |
| 3.1478 | 53.0 | 94711 | 3.2027 | 0.4303 |
| 3.1423 | 54.0 | 96498 | 3.1959 | 0.4312 |
| 3.1383 | 55.0 | 98285 | 3.1911 | 0.4340 |
| 3.1336 | 56.0 | 100072 | 3.1914 | 0.4320 |
| 3.129 | 57.0 | 101859 | 3.1855 | 0.4312 |
| 3.1233 | 58.0 | 103646 | 3.1570 | 0.4337 |
| 3.1198 | 59.0 | 105433 | 3.2042 | 0.4307 |
| 3.1153 | 60.0 | 107220 | 3.1370 | 0.4390 |
| 3.1122 | 61.0 | 109007 | 3.1612 | 0.4412 |
| 3.1093 | 62.0 | 110794 | 3.1642 | 0.4348 |
| 3.1048 | 63.0 | 112581 | 3.1807 | 0.4326 |
| 3.1013 | 64.0 | 114368 | 3.1449 | 0.4359 |
| 3.0977 | 65.0 | 116155 | 3.1408 | 0.4380 |
| 3.0926 | 66.0 | 117942 | 3.1723 | 0.4365 |
| 3.0901 | 67.0 | 119729 | 3.1473 | 0.4380 |
| 3.0882 | 68.0 | 121516 | 3.1401 | 0.4378 |
| 3.0839 | 69.0 | 123303 | 3.1281 | 0.4374 |
| 3.0794 | 70.0 | 125090 | 3.1356 | 0.4367 |
| 3.0766 | 71.0 | 126877 | 3.1019 | 0.4397 |
| 3.074 | 72.0 | 128664 | 3.1626 | 0.4355 |
| 3.0702 | 73.0 | 130451 | 3.1287 | 0.4387 |
| 3.0676 | 74.0 | 132238 | 3.1366 | 0.4379 |
| 3.0648 | 75.0 | 134025 | 3.1782 | 0.4346 |
| 3.0624 | 76.0 | 135812 | 3.1229 | 0.4427 |
| 3.0575 | 77.0 | 137599 | 3.1139 | 0.4430 |
| 3.0549 | 78.0 | 139386 | 3.0948 | 0.4431 |
| 3.052 | 79.0 | 141173 | 3.1030 | 0.4452 |
| 3.0527 | 80.0 | 142960 | 3.0929 | 0.4448 |
| 3.0466 | 81.0 | 144747 | 3.0888 | 0.4428 |
| 3.0439 | 82.0 | 146534 | 3.1035 | 0.4414 |
| 3.0409 | 83.0 | 148321 | 3.1112 | 0.4411 |
| 3.041 | 84.0 | 150108 | 3.1296 | 0.4399 |
| 3.0379 | 85.0 | 151895 | 3.1224 | 0.4428 |
| 3.0332 | 86.0 | 153682 | 3.1101 | 0.4398 |
| 3.0315 | 87.0 | 155469 | 3.1045 | 0.4423 |
| 3.0302 | 88.0 | 157256 | 3.0913 | 0.4446 |
| 3.0265 | 89.0 | 159043 | 3.0745 | 0.4447 |
| 3.0243 | 90.0 | 160830 | 3.0942 | 0.4443 |
| 3.0222 | 91.0 | 162617 | 3.0821 | 0.4432 |
| 3.021 | 92.0 | 164404 | 3.0616 | 0.4473 |
| 3.0183 | 93.0 | 166191 | 3.1021 | 0.4450 |
| 3.0155 | 94.0 | 167978 | 3.1163 | 0.4422 |
| 3.0132 | 95.0 | 169765 | 3.0645 | 0.4493 |
| 3.0118 | 96.0 | 171552 | 3.0922 | 0.4420 |
| 3.0105 | 97.0 | 173339 | 3.1187 | 0.4423 |
| 3.0063 | 98.0 | 175126 | 3.1061 | 0.4462 |
| 3.0035 | 99.0 | 176913 | 3.1098 | 0.4424 |
| 3.0025 | 100.0 | 178700 | 3.0856 | 0.4454 |
| 3.0001 | 101.0 | 180487 | 3.0584 | 0.4504 |
| 2.9979 | 102.0 | 182274 | 3.0897 | 0.4435 |
| 2.9963 | 103.0 | 184061 | 3.0712 | 0.4437 |
| 2.9944 | 104.0 | 185848 | 3.0853 | 0.4458 |
| 2.9931 | 105.0 | 187635 | 3.0809 | 0.4475 |
| 2.992 | 106.0 | 189422 | 3.0910 | 0.4426 |
| 2.9886 | 107.0 | 191209 | 3.0693 | 0.4490 |
| 2.986 | 108.0 | 192996 | 3.0906 | 0.4445 |
| 2.9834 | 109.0 | 194783 | 3.0320 | 0.4538 |
| 2.9829 | 110.0 | 196570 | 3.0760 | 0.4456 |
| 2.9814 | 111.0 | 198357 | 3.0423 | 0.4504 |
| 2.9795 | 112.0 | 200144 | 3.0411 | 0.4529 |
| 2.979 | 113.0 | 201931 | 3.0784 | 0.4463 |
| 2.9781 | 114.0 | 203718 | 3.0526 | 0.4537 |
| 2.9751 | 115.0 | 205505 | 3.0479 | 0.4512 |
| 2.9749 | 116.0 | 207292 | 3.0545 | 0.4493 |
| 2.9735 | 117.0 | 209079 | 3.0529 | 0.4485 |
| 2.9705 | 118.0 | 210866 | 3.0080 | 0.4581 |
| 2.9698 | 119.0 | 212653 | 3.0271 | 0.4537 |
| 2.9674 | 120.0 | 214440 | 3.0477 | 0.4482 |
| 2.9666 | 121.0 | 216227 | 3.0328 | 0.4558 |
| 2.9664 | 122.0 | 218014 | 3.0689 | 0.4463 |
| 2.9639 | 123.0 | 219801 | 3.0749 | 0.4459 |
| 2.9633 | 124.0 | 221588 | 3.0505 | 0.4489 |
| 2.9618 | 125.0 | 223375 | 3.0256 | 0.4535 |
| 2.9589 | 126.0 | 225162 | 3.0522 | 0.4496 |
| 2.9584 | 127.0 | 226949 | 3.0451 | 0.4530 |
| 2.9589 | 128.0 | 228736 | 3.0654 | 0.4502 |
| 2.9581 | 129.0 | 230523 | 2.9989 | 0.4580 |
| 2.9554 | 130.0 | 232310 | 3.0347 | 0.4508 |
| 2.9565 | 131.0 | 234097 | 3.0586 | 0.4498 |
| 2.9548 | 132.0 | 235884 | 3.0170 | 0.4536 |
| 2.9515 | 133.0 | 237671 | 3.0470 | 0.4492 |
| 2.9499 | 134.0 | 239458 | 3.0339 | 0.4515 |
| 2.9514 | 135.0 | 241245 | 3.0474 | 0.4473 |
| 2.9486 | 136.0 | 243032 | 3.0427 | 0.4493 |
| 2.9483 | 137.0 | 244819 | 3.0336 | 0.4534 |
| 2.9491 | 138.0 | 246606 | 3.0274 | 0.4516 |
| 2.9465 | 139.0 | 248393 | 3.0354 | 0.4539 |
| 2.9447 | 140.0 | 250180 | 3.0139 | 0.4526 |
| 2.9449 | 141.0 | 251967 | 3.0163 | 0.4548 |
| 2.9439 | 142.0 | 253754 | 3.0308 | 0.4534 |
| 2.9435 | 143.0 | 255541 | 3.0242 | 0.4579 |
| 2.943 | 144.0 | 257328 | 3.0437 | 0.4513 |
| 2.943 | 145.0 | 259115 | 3.0227 | 0.4544 |
| 2.9403 | 146.0 | 260902 | 3.0464 | 0.4478 |
| 2.9407 | 147.0 | 262689 | 3.0718 | 0.4465 |
| 2.9397 | 148.0 | 264476 | 3.0519 | 0.4487 |
| 2.9392 | 149.0 | 266263 | 3.0163 | 0.4558 |
| 2.9377 | 150.0 | 268050 | 3.0159 | 0.4518 |
| 2.9386 | 151.0 | 269837 | 3.0010 | 0.4545 |
| 2.9391 | 152.0 | 271624 | 3.0346 | 0.4530 |
| 2.9364 | 153.0 | 273411 | 3.0039 | 0.4541 |
| 2.9359 | 154.0 | 275198 | 3.0417 | 0.4519 |
| 2.9359 | 155.0 | 276985 | 3.0161 | 0.4544 |
| 2.936 | 156.0 | 278772 | 3.0169 | 0.4534 |
| 2.9329 | 157.0 | 280559 | 3.0594 | 0.4478 |
| 2.9336 | 158.0 | 282346 | 3.0265 | 0.4555 |
| 2.9341 | 159.0 | 284133 | 3.0276 | 0.4542 |
| 2.933 | 160.0 | 285920 | 3.0324 | 0.4524 |
| 2.9325 | 161.0 | 287707 | 3.0249 | 0.4489 |
| 2.932 | 162.0 | 289494 | 3.0444 | 0.4519 |
| 2.9334 | 163.0 | 291281 | 3.0420 | 0.4494 |
| 2.9318 | 164.0 | 293068 | 2.9972 | 0.4541 |
| 2.9316 | 165.0 | 294855 | 2.9973 | 0.4526 |
| 2.9318 | 166.0 | 296642 | 3.0389 | 0.4529 |
| 2.9301 | 167.0 | 298429 | 3.0131 | 0.4557 |
| 2.9291 | 167.88 | 300000 | 3.0067 | 0.4548 |
Framework versions
- Transformers 4.26.0
- Pytorch 1.14.0a0+410ce96
- Datasets 2.8.0
- Tokenizers 0.13.2