simcse-bert-base-uncased-snli-unsup
A sentence embedding model (768-d, [CLS] pooling) trained from bert-base-uncased by unsupervised SimCSE: the same sentence is encoded twice with different dropout masks (p=0.1); the two views are positives, other sentences in the batch are negatives.
Course assignment U2T02 (Data Engineering, UPY), replicating SimCSE (Gao, Yao & Chen, EMNLP 2021). Author: Joel Perez.
Training data
The 165,529 unique sentences (premises and hypotheses) of a 100k-record subset of SNLI (records with annotator consensus only). This is a much smaller and narrower dataset than the one used in the paper (1M Wikipedia sentences for unsupervised; SNLI+MNLI for supervised), so the numbers are lower than the paper's.
Recipe
| setting | value |
|---|---|
| base model | bert-base-uncased |
| pooling at inference | CLS (no MLP; a Linear+Tanh MLP was used on [CLS] during training only) |
| learning rate / batch size / epochs | 3e-05 / 64 / 1 |
| temperature τ | 0.05 |
| dropout | 0.1 |
| max sequence length (train / eval) | 32 / 128 |
| optimizer | AdamW, linear decay, no warm-up, grad-clip 1.0, mixed precision |
| seed (best of 3 by STS-B dev) | 44 (checkpoint at step 500) |
| hardware | Tesla T4 |
Results (STS-B, Spearman ×100, cosine similarity of normalised embeddings, no regressor)
| model | dev | test |
|---|---|---|
| this model | 77.34 | 69.23 |
| bert-base-uncased (mean) | 59.31 | 47.29 |
| SBERT-2019 | 80.77 | 76.98 |
| SimCSE-unsup (paper) | 82.50 | 76.85 |
The checkpoint was selected on the dev set; the test set was evaluated once per finished run.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Joel119/simcse-bert-base-uncased-snli-unsup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0], emb[1]))
Limitations
- Trained on a small SNLI subset (image-caption domain, short sentences): it generalises less well to other domains/long texts than models trained on larger corpora.
- The model is English-only and was only evaluated on STS-B; other STS/SICK/transfer tasks were not tested.
- Sensitive to negation, antonyms and numbers (failure cases are discussed in the project report).
- It inherits the biases of BERT and of SNLI (crowd-sourced captions).
- Not a replacement for the original SimCSE checkpoints trained on the paper's data.