ProtSent-V2 ESM-2 150M

Contrastively fine-tuned ESM-2 150M producing fixed-length protein embeddings where biological similarity maps to embedding proximity. Intended for retrieval, clustering, and nearest-neighbour transfer.

Retrained on a corpus decontaminated against the benchmark test sets. Predecessor: oriel9p/protsent-esm2-150M. Other scale: GrimSqueaker/ProtSent-V2-35M.

Usage

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("GrimSqueaker/ProtSent-V2-150M")
emb = model.encode([
    "MKTLLLTLVVVTIVCLDLGYT",
    "MKTLLLTLVVVTIVCLDLGYN",
    "AGWYRSPQEGLKPVDTFKDIV",
])
print(cos_sim(emb[0], emb[1:]))

Embeddings are mean-pooled over the final layer, dimension 640. Matryoshka heads at 64/128/256 are available by truncating the embedding.

Training data

Three sources, all decontaminated (see below). No DMS/ProteinGym component — unlike the V1 release, which included ProteinGym DMS pairs under a CoSENT loss.

Pfam and AFDB pairs are sampled within clusters; STRING is a fixed 15M-pair subsample (seed 42) of the filtered pair table.

Decontamination

Every source was searched against the benchmark test sequences with MMseqs2 easy-search (corpus as query, 40% identity, 80% coverage, --cov-mode 1) and matching sequences removed before training.

Filter targets: biomap-research/fold_prediction (remote homology) and Synthyra/bernett_gold_ppi (PPI) test splits. The result was verified by semi-joining each training file against the removal lists: zero flagged sequences remained.

SCOPe-40 was not a filter target — it has no train/test split, so filtering against it would remove nearly all domain sequences from the corpus.

Training configuration

Training code: github.com/oriel9p/ProtSent, train_esm2_150m.sh.

Results

SCOPe-40 structural retrieval, test split, self excluded, no-hit queries counted as failures. Restricted to the 1,693 of 2,207 queries that have a non-self same-family protein in the gallery.

Paired bootstrap over queries (10,000 resamples): V2 − V1 R@1 +0.0809 [+0.0602, +0.1022]; V2 − MMseqs2 R@1 +0.0868 [+0.0620, +0.1116], both excluding zero.

Against a maximally sensitive profile search, top-1 is not ours. With filters off, phmmer reaches R@1 0.7525 against this model's 0.7431. The embedding advantage is in ranking depth and MAP (R@10 0.9368 vs 0.8978, MAP 0.7042 vs 0.6067), at one forward pass per sequence and with indexable sub-linear search, rather than an all-vs-all profile comparison.

Remote homology (the task the corpus was filtered against), test split:

Under 3-NN this model scores below V1 here. V1 was trained on a corpus containing sequences at ≥40% identity to this test set; removing them removed that advantage. Under a linear probe the ordering reverses.

Limitations

• Under a trained linear probe on the final layer, this model is roughly neutral to slightly worse than the stock ESM-2 backbone across a 23-task suite. The advantage is in nearest-neighbour geometry, not in information a trained readout can extract.
• The final layer is not the best pooling layer for property prediction. In a layer sweep on remote homology, an intermediate layer (~2/3 depth) scored higher for every model tested, including the stock backbone.
• V2 differs from V1 in more than decontamination: no hard negatives, proportional sampling, no DMS source, larger effective batch. It is not a controlled ablation of filtering alone.
• Only the remote-homology and PPI test sets were decontamination targets; other benchmark test sets were not filtered against.

Citation

Paper: ProtSent: Protein Sentence Transformers