simcse-bert-base-uncased-snli-sup
A sentence embedding model (768-d, [CLS] pooling) trained from bert-base-uncased by supervised SimCSE: (premise, entailment) pairs are positives; other in-batch hypotheses and the contradiction hypotheses are negatives.
Course assignment U2T02 (Data Engineering, UPY), replicating SimCSE (Gao, Yao & Chen, EMNLP 2021). Author: Joel Perez.
Training data
33,351 (premise, entailment hypothesis) pairs from a 100k-record subset of SNLI; for the ~28 % of premises that also have a contradiction hypothesis, that hypothesis is used as a hard negative. This is a much smaller and narrower dataset than the one used in the paper (1M Wikipedia sentences for unsupervised; SNLI+MNLI for supervised), so the numbers are lower than the paper's.
Recipe
| setting | value |
|---|---|
| base model | bert-base-uncased |
| pooling at inference | CLS (no MLP; no MLP at any time) |
| learning rate / batch size / epochs | 5e-05 / 256 / 3 |
| temperature τ | 0.05 |
| dropout | 0.1 |
| max sequence length (train / eval) | 32 / 128 |
| optimizer | AdamW, linear decay, no warm-up, grad-clip 1.0, mixed precision |
| seed (best of 3 by STS-B dev) | 44 (checkpoint at step 80) |
| hardware | Tesla T4 |
Results (STS-B, Spearman ×100, cosine similarity of normalised embeddings, no regressor)
| model | dev | test |
|---|---|---|
| this model | 82.60 | 78.58 |
| bert-base-uncased (mean) | 59.31 | 47.29 |
| SBERT-2019 | 80.77 | 76.98 |
| SimCSE-sup (paper) | 86.20 | 84.25 |
The checkpoint was selected on the dev set; the test set was evaluated once per finished run.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Joel119/simcse-bert-base-uncased-snli-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0], emb[1]))
Limitations
- Trained on a small SNLI subset (image-caption domain, short sentences): it generalises less well to other domains/long texts than models trained on larger corpora.
- The model is English-only and was only evaluated on STS-B; other STS/SICK/transfer tasks were not tested.
- Sensitive to negation, antonyms and numbers (failure cases are discussed in the project report).
- It inherits the biases of BERT and of SNLI (crowd-sourced captions).
- Not a replacement for the original SimCSE checkpoints trained on the paper's data.