simcse-snli-unsup
A sentence-embedding model trained with unsupervised SimCSE (Gao, Yao & Chen, EMNLP 2021) starting from
bert-base-uncased, as part of the course assignment U2T02 (replicating SimCSE on a small SNLI subset).
Trained by: Ivan Esaú Palomo García, Dalila de los Angeles Ku Dzul
It maps a sentence to a 768-d vector (plain [CLS] pooling); compare sentences with cosine similarity.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Dalila-Ku/simcse-snli-unsup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."], normalize_embeddings=True)
print(emb[0] @ emb[1])
Training data
the 165,529 unique sentences (premises and hypotheses) of snli_train_100k.jsonl, a 100,000-record subsample of the official SNLI train split (records without annotator consensus dropped). No labels are used.
Not the paper's data (1M Wikipedia sentences / SNLI+MNLI), so the numbers below are lower than the paper's.
Recipe
Unsupervised SimCSE (Eq. 1): the same sentence is encoded twice with different dropout masks; other sentences in the batch are negatives.
| hyper-parameter | value |
|---|---|
| base encoder | bert-base-uncased |
| pooling during training | cls_mlp (MLP discarded at inference; inference = plain [CLS]) |
| learning rate / batch size / epochs | 3e-05 / 64 / 1 |
| temperature τ | 0.05 |
| dropout (hidden and attention) | 0.1 |
| max sequence length (training) | 32 |
| warmup / weight decay | 0.0 / 0.0 |
| precision | fp16 (loss computed in fp32) |
| seed | 43 |
| checkpoint | best STS-B dev Spearman (step 1500) |
| hardware | Tesla T4 (10.0 min) |
Results (STS-B, cosine similarity, Spearman ×100, no regressor)
| split | Spearman |
|---|---|
| dev (1,500 pairs; used to pick the checkpoint, so optimistic) | 76.76 |
| test (1,379 pairs; evaluated once) | 68.03 |
Alignment (STS-B dev pairs rated ≥ 4/5): 0.3096; uniformity (t=2): -2.8084 (lower is better for both).
Limitations
- Trained on SNLI sentences only (image captions written by crowd-workers: short, simple, concrete). Expect weaker behaviour on long, formal or domain-specific text.
- Trained on a ~165k-sentence subset; the paper uses ~1M sentences / ~314k NLI pairs, so scores are lower than the published SimCSE.
- English only. Evaluated only on STS-B; no claim about retrieval, clustering or classification quality.
- SNLI contains annotation artifacts and social biases from its crowd-sourced captions; these are inherited by the embeddings.
- Cosine similarity values are not calibrated probabilities; use them for ranking.