ProtSent-V2.5 ESM-2 35M
ProtSent-V2 35M plus one more contrastive pass on a fresh draw of the corpus, with a DMS/ProteinGym CoSENT target and a Global Orthogonal Regularization term added.
Mean-pooled ESM-2 35M embeddings, dimension 480, Matryoshka heads at 64/128/256.
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("GrimSqueaker/ProtSent-V2.5-35M")
emb = model.encode(["MKTLLLTLVVVTIVCLDLGYT", "MKTLLLTLVVVTIVCLDLGYN", "AGWYRSPQEGLKPVDTFKDIV"])
print(cos_sim(emb[0], emb[1:]))
Training
| V2 | V2.5 | |
|---|---|---|
| init | ESM-2 35M | ProtSent-V2 35M |
| loss | CachedMNRL + Matryoshka | + GOR (weight 0.1) + DMS CoSENT |
| pairs | 34.8M | 15.3M — Pfam 285k, AFDB 7M, STRING 7M, DMS 1M |
| cluster sample | k=10, seed 42 | k=5, seed 13 (fresh draw, ~28% rows new) |
| batch / mini-batch | 1024 / 256 | 1024 / 256, CoSENT capped at 256 |
| LR | 2e-4, 3 cosine cycles | 5e-5, half-cosine to zero, 200 warmup |
| max sequence length | 512 | 512 |
| steps / hardware | 4,850 on 7xB300 | 14,924 on 1xB300, 11 h 49 m |
Corpora are decontaminated with MMseqs2 at 40% identity / 80% coverage against the remote-homology and PPI test splits. The DMS parquet is not decontaminated — four suite tasks are DMS-derived (Stability, Fluorescence, beta-lactamase, Variant Effect), and only exact-match overlap has been checked (zero on all three tested).
Results
SCOPe-40 structural retrieval, test split, self excluded, restricted to the 1,693 of 2,207 queries that have a non-self same-family protein in the gallery.
| model | R@1 | R@10 | R@30 | MAP |
|---|---|---|---|---|
| ESM-2 35M | 0.4991 | 0.7614 | 0.8340 | 0.4210 |
| ProtSent-V1 35M | 0.5854 | 0.8511 | 0.9256 | 0.5509 |
| ProtSent-V2 35M | 0.6852 | 0.9220 | 0.9634 | 0.6459 |
| ProtSent-V2.5 35M | 0.6899 | 0.9244 | 0.9681 | 0.6521 |
Paired bootstrap over queries, 10,000 resamples, V2.5 − V2: R@1 +0.0077 [−0.0041, +0.0201], R@10 +0.0018 [−0.0053, +0.0089], R@30 +0.0047 [+0.0000, +0.0100], MAP +0.0078 [+0.0032, +0.0125]. MAP is the only metric excluding zero, so the supportable claim is ranking depth, not top-1. Profile alignment (HMMER) still leads at R@1.
23-task downstream suite. Win/tie/loss over the 20 tasks with a defined one-vs-rest AUC (ties = |delta| < 0.005), with the median delta:
| comparison | k-NN probe | linear probe |
|---|---|---|
| V2 vs ESM-2 35M | 10W/3T/7L, +0.0041 | 2W/7T/11L, −0.0107 |
| V2.5 vs ESM-2 35M | 9W/7T/4L, +0.0046 | 4W/4T/12L, −0.0103 |
| V2.5 vs V2 | 7W/8T/5L, +0.0010 | 7W/8T/5L, +0.0013 |
V2.5 is indistinguishable from V2 on this aggregate. A sign test resolves almost none of these records, so no inferential claim is drawn from the tallies. The linear-probe deficit against vanilla ESM-2 is unchanged.
Largest per-task moves, V2.5 − V2, linear probe: Stability +0.0946, AAV Fitness +0.0732, Fluorescence +0.0149, Variant Effect +0.0128, beta-lactamase +0.0110, Optimal pH −0.0247, Binary Subcellular Localization −0.0128. The gains are the DMS-derived tasks, matching the re-added CoSENT target.
An ablation trained with --gor_weight 0 and everything else identical matches this model
within noise (2W/15T/2L on the k-NN suite; SCOPe-40 eligible R@1 0.6923 / MAP 0.6528). GOR
cost +11.7% per step and is not the source of the improvement over V2.
Training code and full run log:
github.com/oriel9p/ProtSent
(train_esm2_35m_v2p5.sh, RUNS.md).