fedcsis-slot_baseline-xlm_r-es
This model is a fine-tuned version of xlm-roberta-base on the leyzer-fedcsis dataset.
Result on test set:
- Precision: 0.9696
- Recall: 0.9686
- F1: 0.9691
- Accuracy: 0.9904
It achieves the following results on the evaluation set:
- Loss: 0.0521
- Precision: 0.9728
- Recall: 0.9711
- F1: 0.9720
- Accuracy: 0.9914
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
Training results
| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|---|---|---|
| 0.7183 | 1.0 | 941 | 0.1287 | 0.9389 | 0.9429 | 0.9409 | 0.9802 |
| 0.0792 | 2.0 | 1882 | 0.0698 | 0.9551 | 0.9609 | 0.9580 | 0.9876 |
| 0.0502 | 3.0 | 2823 | 0.0586 | 0.9623 | 0.9624 | 0.9624 | 0.9886 |
| 0.0312 | 4.0 | 3764 | 0.0511 | 0.9697 | 0.9668 | 0.9682 | 0.9904 |
| 0.0229 | 5.0 | 4705 | 0.0494 | 0.9715 | 0.9687 | 0.9701 | 0.9913 |
| 0.021 | 6.0 | 5646 | 0.0447 | 0.9697 | 0.9680 | 0.9689 | 0.9911 |
| 0.0139 | 7.0 | 6587 | 0.0512 | 0.9715 | 0.9691 | 0.9703 | 0.9915 |
| 0.0126 | 8.0 | 7528 | 0.0507 | 0.9713 | 0.9699 | 0.9706 | 0.9913 |
| 0.01 | 9.0 | 8469 | 0.0500 | 0.9720 | 0.9702 | 0.9711 | 0.9913 |
| 0.0072 | 10.0 | 9410 | 0.0521 | 0.9728 | 0.9711 | 0.9720 | 0.9914 |
Per slot evaluation on test set
| slot_name | precision | recall | f1 | tc_size |
|---|---|---|---|---|
| album | 0.9500 | 0.9135 | 0.9314 | 104 |
| all_lang | 0.7500 | 1.0000 | 0.8571 | 3 |
| artist | 0.9556 | 0.9685 | 0.9620 | 222 |
| av_alias | 1.0000 | 1.0000 | 1.0000 | 18 |
| caption | 0.9565 | 0.9362 | 0.9462 | 47 |
| category | 0.9091 | 1.0000 | 0.9524 | 10 |
| channel | 0.7857 | 0.7857 | 0.7857 | 14 |
| channel_id | 0.9500 | 1.0000 | 0.9744 | 19 |
| count | 1.0000 | 1.0000 | 1.0000 | 8 |
| date | 0.9762 | 0.9762 | 0.9762 | 42 |
| date_day | 1.0000 | 1.0000 | 1.0000 | 6 |
| date_month | 1.0000 | 1.0000 | 1.0000 | 7 |
| device_name | 0.9770 | 1.0000 | 0.9884 | 85 |
| 1.0000 | 0.9740 | 0.9868 | 192 | |
| event_name | 1.0000 | 1.0000 | 1.0000 | 35 |
| file_name | 1.0000 | 1.0000 | 1.0000 | 10 |
| file_size | 1.0000 | 1.0000 | 1.0000 | 2 |
| filter | 1.0000 | 1.0000 | 1.0000 | 15 |
| hashtag | 1.0000 | 0.9565 | 0.9778 | 46 |
| img_query | 0.9843 | 0.9843 | 0.9843 | 764 |
| label | 1.0000 | 1.0000 | 1.0000 | 7 |
| location | 0.9753 | 0.9875 | 0.9814 | 80 |
| 1.0000 | 1.0000 | 1.0000 | 5 | |
| message | 0.9577 | 0.9607 | 0.9592 | 636 |
| mime_type | 1.0000 | 1.0000 | 1.0000 | 1 |
| name | 0.9677 | 0.9677 | 0.9677 | 31 |
| percent | 0.8571 | 1.0000 | 0.9231 | 6 |
| phone_number | 0.9429 | 0.9763 | 0.9593 | 169 |
| phone_type | 1.0000 | 0.6667 | 0.8000 | 3 |
| picture_url | 1.0000 | 0.9286 | 0.9630 | 42 |
| playlist | 0.9701 | 0.9630 | 0.9665 | 135 |
| portal | 1.0000 | 0.9940 | 0.9970 | 168 |
| priority | 1.0000 | 1.0000 | 1.0000 | 3 |
| purpose | 0.0000 | 0.0000 | 0.0000 | 1 |
| query | 0.9259 | 0.8929 | 0.9091 | 28 |
| rating | 1.0000 | 1.0000 | 1.0000 | 3 |
| review_count | 0.7500 | 0.7500 | 0.7500 | 4 |
| section | 1.0000 | 1.0000 | 1.0000 | 134 |
| seek_time | 1.0000 | 1.0000 | 1.0000 | 2 |
| sender | 0.0000 | 0.0000 | 0.0000 | 1 |
| sender_address | 1.0000 | 1.0000 | 1.0000 | 6 |
| song | 0.9314 | 0.9628 | 0.9468 | 296 |
| src_lang | 0.9872 | 1.0000 | 0.9935 | 77 |
| status | 0.8462 | 0.9565 | 0.8980 | 23 |
| subject | 0.9555 | 0.9567 | 0.9561 | 785 |
| text | 0.9798 | 0.9798 | 0.9798 | 99 |
| time | 1.0000 | 1.0000 | 1.0000 | 32 |
| to | 0.9760 | 0.9651 | 0.9705 | 802 |
| topic | 1.0000 | 1.0000 | 1.0000 | 1 |
| translator | 1.0000 | 1.0000 | 1.0000 | 52 |
| trg_lang | 0.9886 | 1.0000 | 0.9943 | 87 |
| txt_query | 1.0000 | 0.8947 | 0.9444 | 19 |
| username | 1.0000 | 1.0000 | 1.0000 | 6 |
| value | 0.9318 | 0.9535 | 0.9425 | 43 |
| weight | 1.0000 | 1.0000 | 1.0000 | 1 |
Framework versions
- Transformers 4.27.4
- Pytorch 1.13.1+cu116
- Datasets 2.11.0
- Tokenizers 0.13.2
Citation
If you use this model, please cite the following:
@inproceedings{kubis2023caiccaic,
author={Marek Kubis and Paweł Skórzewski and Marcin Sowański and Tomasz Ziętkiewicz},
pages={1319–1324},
title={Center for Artificial Intelligence Challenge on Conversational AI Correctness},
booktitle={Proceedings of the 18th Conference on Computer Science and Intelligence Systems},
year={2023},
doi={10.15439/2023B6058},
url={http://dx.doi.org/10.15439/2023B6058},
volume={35},
series={Annals of Computer Science and Information Systems}
}