modernbert-zeroshot-xnli-eng-0.1

This model is a fine-tuned version of answerdotai/ModernBERT-base on 10% of the english subset of facebook/xnli dataset. It achieves the following results on the evaluation set:

Test Loss: 0.3539
F1: 0.8596

Model description

answerdotai/ModernBERT-base

Intended uses & limitations

Training and evaluation data

10% of the english subset of facebook/xnli dataset.

Training procedure

trained on a single gpu for apx. 20 mins.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 32
eval_batch_size: 16
seed: 42
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
1.0801	0.1629	200	0.8987	0.5868	0.5820	0.6637	0.5868
0.6737	0.3257	400	0.4906	0.8184	0.8181	0.8340	0.8184
0.5361	0.4886	600	0.3931	0.8723	0.8724	0.8759	0.8723
0.4933	0.6515	800	0.3664	0.8782	0.8786	0.8853	0.8782
0.4728	0.8143	1000	0.4300	0.8303	0.8306	0.8604	0.8303
0.4434	0.9772	1200	0.3210	0.8922	0.8923	0.8925	0.8922
0.2859	1.1401	1400	0.3657	0.8483	0.8502	0.8651	0.8483
0.2768	1.3029	1600	0.4162	0.8403	0.8397	0.8520	0.8403
0.258	1.4658	1800	0.4072	0.8543	0.8543	0.8634	0.8543
0.2657	1.6287	2000	0.3763	0.8463	0.8460	0.8537	0.8463
0.2721	1.7915	2200	0.3940	0.8463	0.8464	0.8595	0.8463
0.2878	1.9544	2400	0.3539	0.8603	0.8596	0.8641	0.8603
0.1366	2.1173	2600	0.7444	0.8343	0.8371	0.8738	0.8343

Framework versions

Transformers 4.48.0.dev0
Pytorch 2.4.1+cu121
Datasets 3.1.0
Tokenizers 0.21.0

kaixkhazaki/modernbert-zeroshot-xnli-eng-0.1