karimepachecog/simcse-unsup-snli100k
Sentence embedding model trained with SimCSE (Gao, Yao and Chen, 2021) from bert-base-uncased for U2T02.
Training data
A shared 100k-record subset of SNLI (snli_train_100k.jsonl), not the paper's Wikipedia or SNLI+MNLI training set.
- Mode:
unsupervised - Examples used: 165529 (1.0 of the constructed set)
- Unsupervised set: 165,529 unique SNLI sentences
- Supervised set: 33,351 entailment pairs; a contradiction hard negative exists for 9,488 of them (28.4%) and is masked out otherwise
Recipe
| pooling at train | CLS + BERT pooler MLP |
| pooling at test | cls_before_pooler |
| batch size | 64 |
| learning rate | 3e-05 |
| temperature | 0.05 |
| epochs | 1 |
| max length | 32 |
| dropout | 0.1 |
| seed | 43 |
| hard negatives | False |
| hardware | cuda:Tesla T4 (14.6 GB) |
Optimizer: AdamW, weight decay 0.0, warmup ratio 0.0, linear decay.
Metrics
STS-B Spearman x100, cosine similarity, no regressor.
| split | Spearman |
|---|---|
| dev | 77.49070083546393 |
| test | 69.56246353685135 |
Alignment (STS-B test, gold >= 4): 0.2279043048620224
Uniformity (STS-B test, t=2): -2.502563714981079
Limitations
- English only, and the encoder is uncased.
- Trained on SNLI captions, which are shorter and more concrete than the Wikipedia sentences used for unsupervised SimCSE in the paper.
- The supervised set is about an order of magnitude smaller than SNLI+MNLI, and most pairs have no contradiction hard negative.
- The unsupervised batch size is 64, the same as the paper's BERT-base setting.
- Not a substitute for the published SimCSE checkpoints if you need the paper's STS-B numbers.