SapBERT-PubMedBERT-BabelSynonyms-v2025-01-23

SapBERT-PubMedBERT-BabelSynonyms-2025-01-23 is a biomedical entity representation model fine-tuned by RENCI with biomedical entity synonym pairs extracted from the Babel pipeline (release 2025-01-23) using cambridgeltl/SapBERT-from-PubMedBERT-fulltext as the base model.

The Babel pipelines (preprint) combine identifier cross-reference information from ontologies and biomedical databases to produce sets of equivalent identifiers ("cliques") across a variety of biomedical data types. These are described using the Biolink Model and used for node normalization of knowledge graphs produced by the NCATS Biomedical Data Translator. Babel collects synonym information on all the identifiers it determines to be equivalent, and writes them out as a list of synonyms for each combined identifier clique. Randomly selected pairs of these synonyms were used to fine-tune this SapBERT model.

This model can be used to improve biomedical concept normalization and linking by learning embeddings that place synonymous biomedical concepts closer together in the embedding space. It can also be used for semantic search over biomedical terminology.

Expected input and output

Similar to the SapBERT-PubMedBERT model, the expected input should be a string of biomedical entity names, e.g., "asthma" or "heart failure," and the embedding of the last layer is the output. The dimensions of the output embedding vectors are 768.

Extracting embeddings

The sample embedding script for cambridgeltl/SapBERT-from-PubMedBERT-fulltext works with this fine-tuned model as a drop-in replacement. Just change the model name from "cambridgeltl/SapBERT-from-PubMedBERT-fulltext" to "RenaissanceComputing/SapBERT-PubMedBERT-BabelSynonyms-v2025-01-23".

Code

This model was trained with release v1.0.0 of RENCI-NER/sapbert, a fork of cambridgeltl/sapbert. See that release for the training data preparation, training, evaluation, and inference code used for this model.

Citation

If you use this model, please consider citing the following works:

• The original SapBERT paper which describes the model architecture and training framework:

  @inproceedings{liu-etal-2021-self,
    title = "Self-Alignment Pretraining for Biomedical Entity Representations",
    author = "Liu, Fangyu  and
      Shareghi, Ehsan  and
      Meng, Zaiqiao  and
      Basaldella, Marco  and
      Collier, Nigel",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.naacl-main.334",
    pages = "4228--4238",
    abstract = "Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBERT, SciBERTand and PubMedBERT, our pretraining scheme proves to be both effective and robust.",
  }
  

• The Babel paper

  @misc{morris2026ifairtranslatinginteroperability,
      title={The "I" in FAIR: Translating from Interoperability in Principle to Interoperation in Practice}, 
      author={Evan Morris and Gaurav Vaidya and Phil Owen and Jason Reilly and Karamarie Fecho and Patrick Wang and Yaphet Kebede and E. Kathleen Carter and Chris Bizon},
      year={2026},
      eprint={2601.10008},
      archivePrefix={arXiv},
      primaryClass={cs.DB},
      url={https://arxiv.org/abs/2601.10008}, 
  }