D2C SapModernBERT-XL

SapModernBERT-XL is the biomedical concept retrieval encoder made available to students in UAB's INFO 662/762 course for teaching purposes. It fine-tunes BioClinical-ModernBERT-base with SapBERT-style synonym self-alignment, mapping mention surface strings and concept names into a shared, 768-dimensional embedding space.

The retrieval contract is CLS pooling, L2 normalization, a maximum input length of 32 tokens, and no mention markers.

Intended use

Use the encoder to retrieve biomedical concepts from a vocabulary whose names have been embedded with the same checkpoint, tokenizer, pooling, normalization, and truncation settings. It supports terminology matching, candidate generation, and biomedical entity-linking research.

Usage

Use PyTorch and Transformers 5.5.4, the version tested with this checkpoint and its v5 tokenizer export. The example uses the release repository ID; a local checkpoint directory can be supplied instead. Pin the immutable Hub commit SHA when using this model in the course.

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "UAB-NLP/sapmodernbert-xl"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(
    model_id, attn_implementation="sdpa"
).to(device).eval()

@torch.inference_mode()
def embed(names):
    inputs = tokenizer(
        names, padding=True, truncation=True, max_length=32,
        return_tensors="pt",
    ).to(device)
    cls = model(**inputs).last_hidden_state[:, 0, :]
    return F.normalize(cls.float(), p=2, dim=-1)

queries = embed(["heart attack"])
candidates = embed(["myocardial infarction", "asthma"])
scores = queries @ candidates.T  # cosine similarity, shape [1, 2]
print(scores.cpu())

Training data

Alignment sources

The alignment corpus was assembled from UMLS 2026AA English names and synonyms and handwritten modifier cues. UMLS atom selection used LAT=ENG, nonempty STR, and SUPPRESS != 'O'.

These selections overlap. For preselected CUIs, eligible synonyms were collected across UMLS source vocabularies; HPO-only additions outside that set contributed HPO atoms. The model was not restricted to MeSH or SNOMED CT names.

Corpus size and effective training signal

The retained union contains 2,029,747 entries. Pair construction strips whitespace, deduplicates synonyms case-insensitively, skips entries with fewer than two distinct names, and samples at most 15 unordered positive pairs per class.

Semantic coverage

Joining pair-contributing UMLS CUIs to UMLS 2026AA MRSTY identifies 126 semantic types across all 15 official semantic groups. This is representation, not exhaustive coverage of every concept in those groups. Coverage is concentrated in the primary selections above.

The table uses the official NLM semantic-group mapping. In particular, T034 belongs to PHEN, although the project's routing map includes it under DISO. The four explicitly selected procedure types are a subset of the seven official PROC types observed in the effective training set. A concept may have multiple semantic types.

MedMentions split exposure

The builder reads the full MedMentions annotation file without a train/development/test filter. Concept identities from held-out splits could therefore affect vocabulary selection, although their annotated mention strings and contexts were not used as alignment inputs. MedMentions results should not be presented as evaluation with an unseen concept inventory.

Training procedure

The retained trainer implements a SapBERT-style, rather than a byte-for-byte reproduction of the original SapBERT, objective. It uses L2-normalized CLS embeddings, within-class positive pairs, in-batch other-class negatives, and Multi-Similarity loss with hard-pair mining.

The retained implementation fixes the MS-loss parameters at alpha=1, beta=60, base=0.5, uses mining epsilon 0.1, and uses AdamW with zero weight decay. Pair sampling uses Python random seed 33; epoch shuffling uses 33 + the zero-based epoch number.

Evaluation

The completed record contains an intrinsic synonym-retrieval sanity check over 300 queries and 2,300 candidate names:

This check samples names from the alignment corpus. The implementation does not remove those concepts or pairs from training, so these scores are not held-out benchmark performance. They measure alignment behavior on a small corpus-derived sample, not end-to-end clinical entity-linking accuracy. Downstream performance also depends on vocabulary coverage, mention detection, candidate retrieval, and reranking; scores from those complete systems should not be attributed to the encoder alone.

Limitations

• Alignment is English-only and reflects the source vocabularies and concept-selection recipe.
• Surface strings can be ambiguous. Context-sensitive distinctions, including modifier meanings, are not established by clustering names or grouping cue words into six broad classes.

License and release contents

Model weights are released under MIT, consistent with the base checkpoint's stated license. The upstream MIT copyright and permission notice is preserved in LICENSE. The custom synonym alignment was developed by the UAB NLP Diffuse2Concept project. The encoder release consists of weights, tokenizer, configuration, license, and documentation. UMLS source files, synonym-pair dumps, concept catalogs, and FAISS indices are separate artifacts and are not included. The model license does not replace the terms applicable to external terminology resources used to construct a candidate vocabulary.

References

• BioClinical-ModernBERT-base: upstream model and pretraining details.
• MedMentions: source corpus and full/st21pv variants.