nyt-ingredient-tagger-jina-embeddings-v2-small-en
This model is a fine-tuned version of jinaai/jina-embeddings-v2-small-en on the nyt_ingredients dataset. It achieves the following results on the evaluation set:
- Loss: 0.9890
- Comment: {'precision': 0.4891238056515552, 'recall': 0.6700083542188805, 'f1': 0.5654524089306698, 'number': 7182}
- Name: {'precision': 0.7393011781290907, 'recall': 0.7889533634214485, 'f1': 0.7633206840983521, 'number': 9306}
- Qty: {'precision': 0.9253731343283582, 'recall': 0.9613688009624382, 'f1': 0.943027601127647, 'number': 7481}
- Range End: {'precision': 0.5454545454545454, 'recall': 0.5121951219512195, 'f1': 0.5283018867924528, 'number': 82}
- Unit: {'precision': 0.9031507061927674, 'recall': 0.9693486590038314, 'f1': 0.9350795436284751, 'number': 6003}
- Overall Precision: 0.7401
- Overall Recall: 0.8387
- Overall F1: 0.7863
- Overall Accuracy: 0.7817
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5
- label_smoothing_factor: 0.1
Training results
| Training Loss | Epoch | Step | Validation Loss | Comment | Name | Qty | Range End | Unit | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1.1585 | 0.2 | 1000 | 1.1247 | {'precision': 0.38455309241826097, 'recall': 0.5557343475716794, 'f1': 0.454561770864493, 'number': 6836} | {'precision': 0.6500338458563002, 'recall': 0.7587763855965685, 'f1': 0.7002083333333333, 'number': 8859} | {'precision': 0.8947789025039957, 'recall': 0.9468639887244539, 'f1': 0.9200849140587551, 'number': 7095} | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 74} | {'precision': 0.8575376112987412, 'recall': 0.9760615062030403, 'f1': 0.912968864917872, 'number': 5723} | 0.6666 | 0.7984 | 0.7266 | 0.7220 |
| 1.1018 | 0.4 | 2000 | 1.0677 | {'precision': 0.3960273712222011, 'recall': 0.6095669982445875, 'f1': 0.48012443829934326, 'number': 6836} | {'precision': 0.7094423320659062, 'recall': 0.7582119878090078, 'f1': 0.733016860369946, 'number': 8859} | {'precision': 0.8986009327115256, 'recall': 0.9505285412262157, 'f1': 0.9238356164383562, 'number': 7095} | {'precision': 0.2, 'recall': 0.02702702702702703, 'f1': 0.047619047619047616, 'number': 74} | {'precision': 0.8794009877329935, 'recall': 0.9645290931329722, 'f1': 0.9199999999999999, 'number': 5723} | 0.6853 | 0.8098 | 0.7424 | 0.7415 |
| 1.0676 | 0.59 | 3000 | 1.0472 | {'precision': 0.41734173417341736, 'recall': 0.6104447045055588, 'f1': 0.4957528957528958, 'number': 6836} | {'precision': 0.7021688219122288, 'recall': 0.7784174286036799, 'f1': 0.7383297644539616, 'number': 8859} | {'precision': 0.8949375410913872, 'recall': 0.9592670894996477, 'f1': 0.9259863945578231, 'number': 7095} | {'precision': 0.5161290322580645, 'recall': 0.21621621621621623, 'f1': 0.3047619047619048, 'number': 74} | {'precision': 0.8788844621513944, 'recall': 0.9636554254761489, 'f1': 0.9193198866477745, 'number': 5723} | 0.6939 | 0.8188 | 0.7512 | 0.7541 |
| 1.0613 | 0.79 | 4000 | 1.0459 | {'precision': 0.4413024850042845, 'recall': 0.6026916325336454, 'f1': 0.5095226317091268, 'number': 6836} | {'precision': 0.7297499465697799, 'recall': 0.7708544982503669, 'f1': 0.7497392545424604, 'number': 8859} | {'precision': 0.9064651100013497, 'recall': 0.9465821000704722, 'f1': 0.9260893546607832, 'number': 7095} | {'precision': 0.34210526315789475, 'recall': 0.17567567567567569, 'f1': 0.23214285714285715, 'number': 74} | {'precision': 0.8965631196298744, 'recall': 0.9481041411846933, 'f1': 0.9216135881104034, 'number': 5723} | 0.7177 | 0.8082 | 0.7603 | 0.7502 |
| 1.045 | 0.99 | 5000 | 1.0292 | {'precision': 0.43983577218654596, 'recall': 0.6111761263897015, 'f1': 0.5115396388123661, 'number': 6836} | {'precision': 0.7188987787207618, 'recall': 0.7840614064792866, 'f1': 0.7500674909562118, 'number': 8859} | {'precision': 0.886005680351149, 'recall': 0.9673009161381254, 'f1': 0.9248702917593155, 'number': 7095} | {'precision': 0.3541666666666667, 'recall': 0.22972972972972974, 'f1': 0.27868852459016397, 'number': 74} | {'precision': 0.8777340676632572, 'recall': 0.974663637952123, 'f1': 0.9236628580890875, 'number': 5723} | 0.7080 | 0.8249 | 0.7620 | 0.7610 |
| 1.0334 | 1.19 | 6000 | 1.0344 | {'precision': 0.47399084477736164, 'recall': 0.6664716208308953, 'f1': 0.5539883268482491, 'number': 6836} | {'precision': 0.7198329853862213, 'recall': 0.7784174286036799, 'f1': 0.7479798253701395, 'number': 8859} | {'precision': 0.9296510806611104, 'recall': 0.927554615926709, 'f1': 0.92860166502046, 'number': 7095} | {'precision': 0.36666666666666664, 'recall': 0.2972972972972973, 'f1': 0.3283582089552239, 'number': 74} | {'precision': 0.8982691051600261, 'recall': 0.9612091560370435, 'f1': 0.9286739258884106, 'number': 5723} | 0.7258 | 0.8240 | 0.7718 | 0.7595 |
| 1.0187 | 1.39 | 7000 | 1.0210 | {'precision': 0.4423198816818086, 'recall': 0.6124926857811586, 'f1': 0.5136793031529874, 'number': 6836} | {'precision': 0.7155410238070911, 'recall': 0.7904955412574782, 'f1': 0.751153062318996, 'number': 8859} | {'precision': 0.8767850372804247, 'recall': 0.9778717406624383, 'f1': 0.9245735607675907, 'number': 7095} | {'precision': 0.37142857142857144, 'recall': 0.17567567567567569, 'f1': 0.23853211009174313, 'number': 74} | {'precision': 0.8888354957552459, 'recall': 0.9695963655425476, 'f1': 0.927461139896373, 'number': 5723} | 0.7083 | 0.8287 | 0.7638 | 0.7651 |
| 1.0319 | 1.58 | 8000 | 1.0136 | {'precision': 0.46955690149824675, 'recall': 0.6464306612053833, 'f1': 0.5439773496645535, 'number': 6836} | {'precision': 0.7399957428693061, 'recall': 0.7848515633818716, 'f1': 0.7617639003012875, 'number': 8859} | {'precision': 0.8963893249607535, 'recall': 0.9657505285412262, 'f1': 0.9297781396295542, 'number': 7095} | {'precision': 0.45098039215686275, 'recall': 0.3108108108108108, 'f1': 0.368, 'number': 74} | {'precision': 0.8991981672394044, 'recall': 0.9601607548488555, 'f1': 0.9286800743620078, 'number': 5723} | 0.7280 | 0.8305 | 0.7759 | 0.7700 |
| 1.0154 | 1.78 | 9000 | 1.0071 | {'precision': 0.47295907875796833, 'recall': 0.6729081334113517, 'f1': 0.5554884675763797, 'number': 6836} | {'precision': 0.7443054218800128, 'recall': 0.7856417202844564, 'f1': 0.7644151565074134, 'number': 8859} | {'precision': 0.9104236718224613, 'recall': 0.9540521494009866, 'f1': 0.9317274604267033, 'number': 7095} | {'precision': 0.4126984126984127, 'recall': 0.35135135135135137, 'f1': 0.3795620437956204, 'number': 74} | {'precision': 0.8885522959183674, 'recall': 0.9737899702952997, 'f1': 0.9292205085452273, 'number': 5723} | 0.7285 | 0.8370 | 0.7790 | 0.7732 |
| 1.011 | 1.98 | 10000 | 1.0127 | {'precision': 0.4703654417033473, 'recall': 0.6721767115272089, 'f1': 0.5534477566997892, 'number': 6836} | {'precision': 0.742723104808615, 'recall': 0.7863189976295293, 'f1': 0.7638995503892971, 'number': 8859} | {'precision': 0.915743991358358, 'recall': 0.9558844256518675, 'f1': 0.9353837666367836, 'number': 7095} | {'precision': 0.39285714285714285, 'recall': 0.2972972972972973, 'f1': 0.3384615384615385, 'number': 74} | {'precision': 0.901291060630822, 'recall': 0.9636554254761489, 'f1': 0.9314305016044586, 'number': 5723} | 0.7296 | 0.8353 | 0.7789 | 0.7712 |
| 0.9958 | 2.18 | 11000 | 1.0024 | {'precision': 0.4715127701375246, 'recall': 0.6670567583382094, 'f1': 0.5524928818077179, 'number': 6836} | {'precision': 0.7483029845921776, 'recall': 0.7839485269217744, 'f1': 0.7657111356119073, 'number': 8859} | {'precision': 0.9206457791763579, 'recall': 0.9484143763213531, 'f1': 0.9343237989447376, 'number': 7095} | {'precision': 0.35, 'recall': 0.3783783783783784, 'f1': 0.36363636363636365, 'number': 74} | {'precision': 0.8947876447876448, 'recall': 0.9718679014502883, 'f1': 0.9317363263254879, 'number': 5723} | 0.7318 | 0.8334 | 0.7793 | 0.7763 |
| 1.0042 | 2.38 | 12000 | 1.0007 | {'precision': 0.4789602641951635, 'recall': 0.657694558221182, 'f1': 0.5542747950440732, 'number': 6836} | {'precision': 0.7298981923955953, 'recall': 0.7930917710802574, 'f1': 0.7601839329185826, 'number': 8859} | {'precision': 0.9267211525141986, 'recall': 0.9429175475687104, 'f1': 0.9347491965907503, 'number': 7095} | {'precision': 0.35555555555555557, 'recall': 0.43243243243243246, 'f1': 0.3902439024390244, 'number': 74} | {'precision': 0.9089250165892502, 'recall': 0.9573650183470208, 'f1': 0.932516381584546, 'number': 5723} | 0.7333 | 0.8299 | 0.7786 | 0.7769 |
| 1.0048 | 2.57 | 13000 | 0.9943 | {'precision': 0.47168994262206343, 'recall': 0.6373610298420129, 'f1': 0.542151434082001, 'number': 6836} | {'precision': 0.7342649994746243, 'recall': 0.7888023478947963, 'f1': 0.7605572485851111, 'number': 8859} | {'precision': 0.8968563263185243, 'recall': 0.9730796335447498, 'f1': 0.9334144527817211, 'number': 7095} | {'precision': 0.4307692307692308, 'recall': 0.3783783783783784, 'f1': 0.4028776978417266, 'number': 74} | {'precision': 0.8915373540233562, 'recall': 0.9737899702952997, 'f1': 0.9308501753799899, 'number': 5723} | 0.7278 | 0.8343 | 0.7774 | 0.7787 |
| 0.9911 | 2.77 | 14000 | 0.9951 | {'precision': 0.4768817204301075, 'recall': 0.6487712112346401, 'f1': 0.5497025285076846, 'number': 6836} | {'precision': 0.729257190151045, 'recall': 0.7956880009030365, 'f1': 0.7610256410256411, 'number': 8859} | {'precision': 0.9042483230303827, 'recall': 0.9689922480620154, 'f1': 0.9355014287658184, 'number': 7095} | {'precision': 0.5319148936170213, 'recall': 0.33783783783783783, 'f1': 0.4132231404958678, 'number': 74} | {'precision': 0.8965628529933839, 'recall': 0.9708195002621003, 'f1': 0.9322147651006711, 'number': 5723} | 0.7296 | 0.8374 | 0.7798 | 0.7786 |
| 0.9991 | 2.97 | 15000 | 0.9921 | {'precision': 0.4791033832617576, 'recall': 0.6691047396138092, 'f1': 0.5583836904107916, 'number': 6836} | {'precision': 0.7481054541573273, 'recall': 0.7911728186025511, 'f1': 0.7690366469168313, 'number': 8859} | {'precision': 0.9137861466039005, 'recall': 0.9575757575757575, 'f1': 0.935168616655196, 'number': 7095} | {'precision': 0.4166666666666667, 'recall': 0.40540540540540543, 'f1': 0.4109589041095891, 'number': 74} | {'precision': 0.8894720101781171, 'recall': 0.977284640922593, 'f1': 0.9313129631171426, 'number': 5723} | 0.7337 | 0.8395 | 0.7831 | 0.7807 |
| 0.9805 | 3.17 | 16000 | 0.9880 | {'precision': 0.4859154929577465, 'recall': 0.6560854300760679, 'f1': 0.5583219220714553, 'number': 6836} | {'precision': 0.7423922231614539, 'recall': 0.7930917710802574, 'f1': 0.7669049828084921, 'number': 8859} | {'precision': 0.9187102018696653, 'recall': 0.9557434813248766, 'f1': 0.9368610113290964, 'number': 7095} | {'precision': 0.4444444444444444, 'recall': 0.3783783783783784, 'f1': 0.4087591240875913, 'number': 74} | {'precision': 0.9000486775920817, 'recall': 0.9692468984798183, 'f1': 0.9333669863705198, 'number': 5723} | 0.7389 | 0.8349 | 0.7840 | 0.7822 |
| 0.9848 | 3.37 | 17000 | 0.9842 | {'precision': 0.48933174482833314, 'recall': 0.6609128145114102, 'f1': 0.5623249735515589, 'number': 6836} | {'precision': 0.7466623945316672, 'recall': 0.7891409865673327, 'f1': 0.7673142355394577, 'number': 8859} | {'precision': 0.9149737656397148, 'recall': 0.9585623678646934, 'f1': 0.936261013215859, 'number': 7095} | {'precision': 0.4126984126984127, 'recall': 0.35135135135135137, 'f1': 0.3795620437956204, 'number': 74} | {'precision': 0.899171943497321, 'recall': 0.9676742966975362, 'f1': 0.9321663019693655, 'number': 5723} | 0.7403 | 0.8351 | 0.7848 | 0.7824 |
| 0.9771 | 3.56 | 18000 | 0.9834 | {'precision': 0.4883396023643203, 'recall': 0.6647162083089526, 'f1': 0.5630382256365777, 'number': 6836} | {'precision': 0.7373874816830647, 'recall': 0.795236482672988, 'f1': 0.7652202248411449, 'number': 8859} | {'precision': 0.9162388543636855, 'recall': 0.9558844256518675, 'f1': 0.9356418569359177, 'number': 7095} | {'precision': 0.4583333333333333, 'recall': 0.44594594594594594, 'f1': 0.4520547945205479, 'number': 74} | {'precision': 0.8992864093415505, 'recall': 0.968897431417089, 'f1': 0.9327950206072841, 'number': 5723} | 0.7369 | 0.8378 | 0.7841 | 0.7837 |
| 0.9787 | 3.76 | 19000 | 0.9832 | {'precision': 0.4892808110676946, 'recall': 0.677735517846694, 'f1': 0.5682919349892671, 'number': 6836} | {'precision': 0.7466029723991507, 'recall': 0.7938819279828423, 'f1': 0.7695169319984682, 'number': 8859} | {'precision': 0.9206090266449157, 'recall': 0.9544749823819592, 'f1': 0.9372361774271676, 'number': 7095} | {'precision': 0.4189189189189189, 'recall': 0.4189189189189189, 'f1': 0.4189189189189189, 'number': 74} | {'precision': 0.9048244174597965, 'recall': 0.9634806919447843, 'f1': 0.9332317847169331, 'number': 5723} | 0.7399 | 0.8389 | 0.7863 | 0.7844 |
| 0.9746 | 3.96 | 20000 | 0.9827 | {'precision': 0.4950890447922288, 'recall': 0.6710064365125804, 'f1': 0.5697782746413266, 'number': 6836} | {'precision': 0.7460368124268539, 'recall': 0.7915114572750874, 'f1': 0.768101654069449, 'number': 8859} | {'precision': 0.9120629837203096, 'recall': 0.9633544749823819, 'f1': 0.9370073342929603, 'number': 7095} | {'precision': 0.40963855421686746, 'recall': 0.4594594594594595, 'f1': 0.43312101910828027, 'number': 74} | {'precision': 0.9003893575600259, 'recall': 0.9697710990739122, 'f1': 0.9337932194834694, 'number': 5723} | 0.7412 | 0.8402 | 0.7876 | 0.7846 |
| 0.976 | 4.16 | 21000 | 0.9836 | {'precision': 0.4884607241160279, 'recall': 0.6749561146869514, 'f1': 0.5667608401916225, 'number': 6836} | {'precision': 0.7483774869666986, 'recall': 0.7939948075403545, 'f1': 0.7705115565779385, 'number': 8859} | {'precision': 0.9147193123824873, 'recall': 0.9599718111346018, 'f1': 0.936799394814662, 'number': 7095} | {'precision': 0.4125, 'recall': 0.44594594594594594, 'f1': 0.42857142857142855, 'number': 74} | {'precision': 0.9035317200784827, 'recall': 0.9655774943211602, 'f1': 0.9335247909451813, 'number': 5723} | 0.7393 | 0.8402 | 0.7865 | 0.7854 |
| 0.9635 | 4.36 | 22000 | 0.9832 | {'precision': 0.49533612093920876, 'recall': 0.6758338209479228, 'f1': 0.5716760502381983, 'number': 6836} | {'precision': 0.7475583864118895, 'recall': 0.7948978440004515, 'f1': 0.7705016685814322, 'number': 8859} | {'precision': 0.9186408555570597, 'recall': 0.9564482029598309, 'f1': 0.9371633752244165, 'number': 7095} | {'precision': 0.391304347826087, 'recall': 0.4864864864864865, 'f1': 0.43373493975903615, 'number': 74} | {'precision': 0.9026418786692759, 'recall': 0.9671500961034423, 'f1': 0.9337832138338253, 'number': 5723} | 0.7423 | 0.8402 | 0.7882 | 0.7851 |
| 0.9688 | 4.55 | 23000 | 0.9836 | {'precision': 0.4930739135032251, 'recall': 0.6821240491515506, 'f1': 0.5723930522310194, 'number': 6836} | {'precision': 0.7467897697124058, 'recall': 0.7943334462128908, 'f1': 0.7698282463625424, 'number': 8859} | {'precision': 0.9208593962469405, 'recall': 0.9544749823819592, 'f1': 0.9373659076752716, 'number': 7095} | {'precision': 0.35714285714285715, 'recall': 0.5405405405405406, 'f1': 0.4301075268817204, 'number': 74} | {'precision': 0.9021013194331324, 'recall': 0.9676742966975362, 'f1': 0.9337379868487607, 'number': 5723} | 0.7403 | 0.8413 | 0.7876 | 0.7860 |
| 0.9669 | 4.75 | 24000 | 0.9803 | {'precision': 0.4968897468897469, 'recall': 0.677735517846694, 'f1': 0.5733910891089109, 'number': 6836} | {'precision': 0.7478168264110756, 'recall': 0.7926402528502088, 'f1': 0.7695764151460354, 'number': 8859} | {'precision': 0.9173631706659477, 'recall': 0.9591261451726568, 'f1': 0.9377799214497348, 'number': 7095} | {'precision': 0.4, 'recall': 0.4864864864864865, 'f1': 0.43902439024390244, 'number': 74} | {'precision': 0.8998864189518092, 'recall': 0.9690721649484536, 'f1': 0.9331987211845869, 'number': 5723} | 0.7424 | 0.8410 | 0.7886 | 0.7873 |
| 0.9691 | 4.95 | 25000 | 0.9796 | {'precision': 0.4962978860392746, 'recall': 0.6765652428320655, 'f1': 0.5725781491798204, 'number': 6836} | {'precision': 0.7474457215836526, 'recall': 0.7927531324077209, 'f1': 0.7694330320460149, 'number': 8859} | {'precision': 0.9183783783783783, 'recall': 0.9578576462297392, 'f1': 0.9377026560883062, 'number': 7095} | {'precision': 0.4065934065934066, 'recall': 0.5, 'f1': 0.4484848484848485, 'number': 74} | {'precision': 0.9002599090318388, 'recall': 0.968373230822995, 'f1': 0.933075174678003, 'number': 5723} | 0.7423 | 0.8403 | 0.7883 | 0.7871 |
Framework versions
- Transformers 4.34.1
- Pytorch 2.1.0+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1