RusselKuAguilar/simcse-bert-uncased-unsup

Unsupervised SimCSE sentence encoder trained from bert-base-uncased on a 100k-record subset of SNLI, as a course replication of Gao, Yao & Chen (EMNLP 2021). It maps an English sentence to a 768-dimensional vector; compare sentences with cosine similarity.

Team (Universidad Politécnica de Yucatán, U2T02): Acosta Castellanos Bianca Alexandra, Canche Chuc Angel Rivaldo, Ku Aguilar Russel, Sanchez Novelo Damian, Velasco Martin Jonathan Abisai.

Training data

165,528 unique sentences (premises and hypotheses, whitespace-stripped) from snli_train_100k.jsonl, a 100k-record subset of the SNLI train split. Labels are not used: each sentence is its own positive, the two views differ only by dropout.

The paper trained on much more data (1M Wikipedia sentences for unsupervised; ~314k SNLI+MNLI pairs for supervised). This model is therefore not expected to reach the paper's numbers; see Limitations.

Training recipe

Unsupervised SimCSE (Gao et al., 2021, Eq. 1): InfoNCE over a batch of N sentences, positive = the same sentence encoded with an independent dropout mask, negatives = the other N-1 = 63 sentences of the batch.

Inference representation: [CLS] token of the last layer. The MLP head was used only during training and is discarded at inference, as in the paper.

Evaluation (STS-B, Spearman x100)

Protocol: encode both sentences, L2-normalize, cosine similarity, Spearman against the human scores; no regressor ("all" aggregation, Gao et al. Appendix B). Test was evaluated once, after selecting the checkpoint on dev. Alignment uses STS-B pairs with score >= 4; uniformity uses all sentences (Wang & Isola, 2020; lower is better for both).

Reference baselines with the same evaluation code: raw bert-base-uncased (mean pooling) 59.31 / 47.29; SBERT-2019 (bert-base-nli-mean-tokens) 80.77 / 76.98 (dev / test).

Verified after upload: reloaded from the Hub and re-evaluated on STS-B test = 67.32 (table value 67.32).

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("RusselKuAguilar/simcse-bert-uncased-unsup")
sentences = [
    "A man is playing a guitar.",
    "Someone is playing an instrument.",
    "A dog is chasing a ball in the park.",
]
emb = model.encode(sentences, normalize_embeddings=True)
print(emb @ emb.T)  # cosine similarity matrix

Use sentence-transformers (not a raw AutoModel with your own pooling): the pooling is stored in the model's module configuration, and using a different pooling silently changes the embeddings.

Limitations

• Small, narrow training corpus. Only SNLI sentences: short, simple descriptions of photographs (SNLI premises are Flickr30k captions). Quality drops on other genres; on STS-B the test score is clearly below dev (76.32 vs 67.32).
• Topical similarity is over-rated. Sentences that share a template and a topic but differ in the key entity or event (e.g. two different attacks in two different countries) get high cosine similarity even when humans rate them as unrelated.
• English only, max 64 tokens; longer inputs are truncated. Not evaluated on retrieval benchmarks, other languages, or specialised domains (legal, biomedical, code).
• Biases. Inherits the social biases of BERT pre-training data and of SNLI, whose hypotheses were written by crowd workers and contain known annotation artifacts and stereotypes.
• Single seed. Reported from one training seed; run-to-run variation is roughly +-0.5 Spearman on dev in our supervised seed study, so differences under ~1 point should not be over-interpreted.
• Intended for teaching and research, not for decisions about people.

Citation

@inproceedings{gao2021simcse,
  title={SimCSE: Simple Contrastive Learning of Sentence Embeddings},
  author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
  booktitle={EMNLP},
  year={2021}
}