karimepachecog/simcse-sup-snli100k
Sentence embedding model trained with SimCSE (Gao, Yao and Chen, 2021) from bert-base-uncased for U2T02.
Training data
A shared 100k-record subset of SNLI (snli_train_100k.jsonl), not the paper's Wikipedia or SNLI+MNLI training set.
- Mode:
supervised - Examples used: 33351 (1.0 of the constructed set)
- Unsupervised set: 165,529 unique SNLI sentences
- Supervised set: 33,351 entailment pairs; a contradiction hard negative exists for 9,488 of them (28.4%) and is masked out otherwise
Recipe
| pooling at train | CLS + BERT pooler MLP |
| pooling at test | cls |
| batch size | 128 |
| learning rate | 5e-05 |
| temperature | 0.05 |
| epochs | 3 |
| max length | 32 |
| dropout | 0.1 |
| seed | 42 |
| hard negatives | True |
| hardware | cuda:Tesla T4 (14.6 GB) |
Optimizer: AdamW, weight decay 0.0, warmup ratio 0.0, linear decay.
Metrics
STS-B Spearman x100, cosine similarity, no regressor.
| split | Spearman |
|---|---|
| dev | 78.03973124609854 |
| test | 74.73372610133005 |
Alignment (STS-B test, gold >= 4): 0.13581092655658722
Uniformity (STS-B test, t=2): -2.472869873046875
Limitations
- English only, and the encoder is uncased.
- Trained on SNLI captions, which are shorter and more concrete than the Wikipedia sentences used for unsupervised SimCSE in the paper.
- The supervised set is about an order of magnitude smaller than SNLI+MNLI, and most pairs have no contradiction hard negative.
- The supervised batch size is 128, not the paper's 512. The learning rate was retuned on the STS-B dev set at this batch: 1e-5 scored 74.28, 3e-5 scored 76.67, and 5e-5 scored 78.04. This checkpoint uses 5e-05.
- Not a substitute for the published SimCSE checkpoints if you need the paper's STS-B numbers.