simcse-snli-sup
A sentence-embedding model trained with supervised SimCSE (Gao, Yao & Chen, EMNLP 2021) starting from
bert-base-uncased, as part of the course assignment U2T02 (replicating SimCSE on a small SNLI subset).
Trained by: Ivan Esaú Palomo García, Dalila de los Angeles Ku Dzul
It maps a sentence to a 768-d vector (plain [CLS] pooling); compare sentences with cosine similarity.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Dalila-Ku/simcse-snli-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."], normalize_embeddings=True)
print(emb[0] @ emb[1])
Training data
the 33,351 (premise, entailment-hypothesis) pairs of snli_train_100k.jsonl (a 100,000-record subsample of the official SNLI train split, no-consensus records dropped). For 9,488 pairs (28.4 %) a contradiction hypothesis of the same premise is added as hard negative (Eq. 5 of the paper).
Not the paper's data (1M Wikipedia sentences / SNLI+MNLI), so the numbers below are lower than the paper's.
Recipe
Supervised SimCSE (Eq. 5): entailment pairs are positives, other hypotheses in the batch and the contradiction hypotheses are negatives.
| hyper-parameter | value |
|---|---|
| base encoder | bert-base-uncased |
| pooling during training | cls (MLP discarded at inference; inference = plain [CLS]) |
| learning rate / batch size / epochs | 5e-05 / 128 / 3 |
| temperature τ | 0.05 |
| dropout (hidden and attention) | 0.1 |
| max sequence length (training) | 32 |
| warmup / weight decay | 0.0 / 0.0 |
| precision | fp16 (loss computed in fp32) |
| seed | 43 |
| checkpoint | best STS-B dev Spearman (step 50) |
| hardware | Tesla T4 (6.56 min) |
Results (STS-B, cosine similarity, Spearman ×100, no regressor)
| split | Spearman |
|---|---|
| dev (1,500 pairs; used to pick the checkpoint, so optimistic) | 82.27 |
| test (1,379 pairs; evaluated once) | 77.75 |
Alignment (STS-B dev pairs rated ≥ 4/5): 0.1853; uniformity (t=2): -3.0021 (lower is better for both).
Limitations
- Trained on SNLI sentences only (image captions written by crowd-workers: short, simple, concrete). Expect weaker behaviour on long, formal or domain-specific text.
- Trained on a ~33k-pair subset; the paper uses ~1M sentences / ~314k NLI pairs, so scores are lower than the published SimCSE.
- English only. Evaluated only on STS-B; no claim about retrieval, clustering or classification quality.
- Only 28.4 % of the training pairs carry a hard negative.
- SNLI contains annotation artifacts and social biases from its crowd-sourced captions; these are inherited by the embeddings.
- Cosine similarity values are not calibrated probabilities; use them for ranking.