distilbert-base-uncased-distilled-squad_qa_model
This model is a fine-tuned version of distilbert-base-uncased-distilled-squad on the subjqa dataset. It achieves the following results on the evaluation set:
- Loss: 2.9380
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 100
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 4.1556 | 1.0 | 32 | 4.1242 |
| 4.0411 | 2.0 | 64 | 4.0582 |
| 3.9828 | 3.0 | 96 | 3.9948 |
| 3.9068 | 4.0 | 128 | 3.9378 |
| 3.8152 | 5.0 | 160 | 3.8835 |
| 3.7906 | 6.0 | 192 | 3.8329 |
| 3.7543 | 7.0 | 224 | 3.7842 |
| 3.7173 | 8.0 | 256 | 3.7377 |
| 3.6717 | 9.0 | 288 | 3.6958 |
| 3.6219 | 10.0 | 320 | 3.6559 |
| 3.587 | 11.0 | 352 | 3.6185 |
| 3.6111 | 12.0 | 384 | 3.5808 |
| 3.5374 | 13.0 | 416 | 3.5483 |
| 3.4506 | 14.0 | 448 | 3.5175 |
| 3.4286 | 15.0 | 480 | 3.4873 |
| 3.4021 | 16.0 | 512 | 3.4596 |
| 3.432 | 17.0 | 544 | 3.4328 |
| 3.3235 | 18.0 | 576 | 3.4079 |
| 3.3627 | 19.0 | 608 | 3.3841 |
| 3.323 | 20.0 | 640 | 3.3615 |
| 3.3127 | 21.0 | 672 | 3.3389 |
| 3.2635 | 22.0 | 704 | 3.3199 |
| 3.2542 | 23.0 | 736 | 3.3013 |
| 3.2302 | 24.0 | 768 | 3.2846 |
| 3.1699 | 25.0 | 800 | 3.2676 |
| 3.2333 | 26.0 | 832 | 3.2516 |
| 3.2204 | 27.0 | 864 | 3.2364 |
| 3.1809 | 28.0 | 896 | 3.2218 |
| 3.1739 | 29.0 | 928 | 3.2082 |
| 3.1966 | 30.0 | 960 | 3.1950 |
| 3.1513 | 31.0 | 992 | 3.1826 |
| 3.135 | 32.0 | 1024 | 3.1713 |
| 3.1253 | 33.0 | 1056 | 3.1599 |
| 3.0768 | 34.0 | 1088 | 3.1498 |
| 3.1031 | 35.0 | 1120 | 3.1394 |
| 3.064 | 36.0 | 1152 | 3.1293 |
| 3.0391 | 37.0 | 1184 | 3.1200 |
| 3.0701 | 38.0 | 1216 | 3.1117 |
| 3.0787 | 39.0 | 1248 | 3.1032 |
| 3.0423 | 40.0 | 1280 | 3.0956 |
| 3.0214 | 41.0 | 1312 | 3.0875 |
| 3.0289 | 42.0 | 1344 | 3.0804 |
| 2.9667 | 43.0 | 1376 | 3.0736 |
| 3.0341 | 44.0 | 1408 | 3.0671 |
| 3.0098 | 45.0 | 1440 | 3.0606 |
| 3.0202 | 46.0 | 1472 | 3.0544 |
| 2.9598 | 47.0 | 1504 | 3.0490 |
| 2.9734 | 48.0 | 1536 | 3.0430 |
| 2.9381 | 49.0 | 1568 | 3.0375 |
| 2.9444 | 50.0 | 1600 | 3.0328 |
| 2.9357 | 51.0 | 1632 | 3.0280 |
| 2.9453 | 52.0 | 1664 | 3.0237 |
| 2.9906 | 53.0 | 1696 | 3.0191 |
| 2.934 | 54.0 | 1728 | 3.0148 |
| 2.9076 | 55.0 | 1760 | 3.0110 |
| 2.9874 | 56.0 | 1792 | 3.0070 |
| 2.9682 | 57.0 | 1824 | 3.0032 |
| 2.9287 | 58.0 | 1856 | 2.9994 |
| 2.9575 | 59.0 | 1888 | 2.9956 |
| 2.8618 | 60.0 | 1920 | 2.9926 |
| 2.9614 | 61.0 | 1952 | 2.9893 |
| 2.9463 | 62.0 | 1984 | 2.9861 |
| 2.8927 | 63.0 | 2016 | 2.9834 |
| 2.9048 | 64.0 | 2048 | 2.9805 |
| 2.9161 | 65.0 | 2080 | 2.9777 |
| 2.9117 | 66.0 | 2112 | 2.9753 |
| 2.932 | 67.0 | 2144 | 2.9729 |
| 2.9148 | 68.0 | 2176 | 2.9706 |
| 2.8919 | 69.0 | 2208 | 2.9683 |
| 2.9278 | 70.0 | 2240 | 2.9662 |
| 2.869 | 71.0 | 2272 | 2.9643 |
| 2.8844 | 72.0 | 2304 | 2.9622 |
| 2.8636 | 73.0 | 2336 | 2.9603 |
| 2.8734 | 74.0 | 2368 | 2.9585 |
| 2.8934 | 75.0 | 2400 | 2.9569 |
| 2.86 | 76.0 | 2432 | 2.9551 |
| 2.8366 | 77.0 | 2464 | 2.9539 |
| 2.8887 | 78.0 | 2496 | 2.9522 |
| 2.8632 | 79.0 | 2528 | 2.9511 |
| 2.8691 | 80.0 | 2560 | 2.9496 |
| 2.8597 | 81.0 | 2592 | 2.9484 |
| 2.8775 | 82.0 | 2624 | 2.9473 |
| 2.8491 | 83.0 | 2656 | 2.9461 |
| 2.8639 | 84.0 | 2688 | 2.9450 |
| 2.8659 | 85.0 | 2720 | 2.9443 |
| 2.8557 | 86.0 | 2752 | 2.9433 |
| 2.8188 | 87.0 | 2784 | 2.9423 |
| 2.8896 | 88.0 | 2816 | 2.9416 |
| 2.8102 | 89.0 | 2848 | 2.9409 |
| 2.8452 | 90.0 | 2880 | 2.9403 |
| 2.8437 | 91.0 | 2912 | 2.9399 |
| 2.8193 | 92.0 | 2944 | 2.9397 |
| 2.8645 | 93.0 | 2976 | 2.9391 |
| 2.8745 | 94.0 | 3008 | 2.9388 |
| 2.8568 | 95.0 | 3040 | 2.9385 |
| 2.8832 | 96.0 | 3072 | 2.9382 |
| 2.8801 | 97.0 | 3104 | 2.9382 |
| 2.8488 | 98.0 | 3136 | 2.9383 |
| 2.8233 | 99.0 | 3168 | 2.9380 |
| 2.8505 | 100.0 | 3200 | 2.9380 |
Framework versions
- Transformers 4.28.0
- Pytorch 1.13.0a0+d321be6
- Datasets 2.12.0
- Tokenizers 0.13.3