SciEmbed-NoDAPT-CTX
The CTX recipe with Stage 1 (domain-adaptive MLM) skipped — contrastive straight off ModernBERT-base. Shows DAPT adds only ~0.1 once Signals A+B are present.
A 149M-parameter ModernBERT-base scientific document embedder trained with citation-context sentences as the primary contrastive signal. Part of the SciEmbed release (Findings of EMNLP 2026).
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("J0nasW/sciembed-nodapt-ctx")
emb = model.encode(["citation-context supervision for scientific embeddings"],
normalize_embeddings=True)
- Context length: 512 tokens
- Pooling: mean · Output dim: 768 (Matryoshka-truncatable to 512/256/128)
- License: MIT
SciRepEval (4-category macro)
| Classif. | Regr. | Prox. | Search | Overall |
|---|---|---|---|---|
| 75.3 | 28.2 | 80.8 | 82.6 | 66.7 ± 0.07 |
Citation
See the repository README. Paper: SciEmbed: Citation-Context Supervision for Scientific Document Embeddings, Findings of the Association for Computational Linguistics: EMNLP 2026.