simcse-bert-base-snli100k-sup

A supervised SimCSE sentence-embedding model (Gao, Yao & Chen, EMNLP 2021) trained from bert-base-uncased as a course replication project. Team: Yazira Hernandez Rives.

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("yaz12611/simcse-bert-base-snli100k-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."], normalize_embeddings=True)
print(emb[0] @ emb[1])

Training data

33,351 (premise, entailment) pairs from a 100k SNLI subset; 9,488 (28.4%) with a contradiction hard negative. Taken from the SNLI train split (records without annotator consensus removed, 100k sampled). This is much less data than the paper used (1M Wikipedia sentences for unsupervised SimCSE; 275k SNLI+MNLI triplets for supervised SimCSE).

Recipe

Pooling: [CLS] + Linear/tanh MLP (kept at inference). Contrastive InfoNCE loss with in-batch negatives (Eq. 1 / Eq. 5 of the paper). Hyper-parameters: batch 128, lr 5e-5, 300 steps, max_len 32, temperature 0.05, dropout 0.1, AdamW, linear decay, fp16, seed 42. Checkpoint chosen by STS-B dev Spearman.

Evaluation (STS-B, cosine of L2-normalised embeddings, Spearman × 100, no regressor)

The test number was verified by reloading this repository from the Hub and re-running the evaluation.

Limitations

• Trained only on SNLI, whose sentences are short, literal descriptions of Flickr photos. Expect weaker results on long text, technical or news language, and anything far from that domain.
• English only; inputs truncated to 32 tokens during training.
• Like most contrastive sentence encoders it is weak on negation, numbers and word order: sentences with high lexical overlap but opposite meaning can still receive high similarity.
• STS-B is the only benchmark evaluated; no transfer (SentEval) or retrieval benchmarks.
• Inherits biases of BERT and of crowd-written SNLI hypotheses (known annotation artifacts).