Clinical ModernBERT Embedding Model
A specialized medical embedding model fine-tuned from Clinical ModernBERT using InfoNCE contrastive learning on PubMed title-abstract pairs.
Model Details
- Base Model: thomas-sounack/BioClinical-ModernBERT-base
- Training Method: InfoNCE contrastive learning
- Training Data: PubMed title-abstract pairs
- Max Sequence Length: 2048 tokens
Usage
from sentence_transformers import SentenceTransformer
# Load the model
model = SentenceTransformer("lokeshch19/ModernPubMedBERT")
# Encode medical texts
texts = [
"Rheumatoid arthritis is an autoimmune disorder attacking joint linings.",
"Inflammatory cytokines in RA lead to progressive cartilage and bone destruction."
]
embeddings = model.encode(texts)
Applications
- Medical document similarity analysis
- Clinical text retrieval systems
- Biomedical literature search
- Medical concept matching and classification
Model Comparison
Compared to NeuML/bioclinical-modernbert-base-embeddings, our model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content.
Comprehensive Evaluation Results
| Metric | Our Model | NeuML Model | Improvement |
|---|---|---|---|
| Accuracy@1 | 91.28% | 85.86% | +6.3% |
| Accuracy@3 | 98.46% | 95.66% | +2.9% |
| Accuracy@5 | 99.24% | 97.14% | +2.2% |
| Accuracy@10 | 99.64% | 98.29% | +1.4% |
| NDCG@5 | 95.96% | 92.37% | +3.9% |
| NDCG@10 | 96.10% | 92.75% | +3.6% |
| MRR@10 | 94.89% | 90.90% | +4.4% |
| MAP@100 | 94.91% | 90.96% | +4.3% |
Evaluation performed using InformationRetrievalEvaluator from sentence-transformers on the gamino/wiki_medical_terms dataset.
Model Comparison
Compared to NeuML/bioclinical-modernbert-base-embeddings, this model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content.
Medical Text Similarity
Example 1: Related Medical Concepts
text1 = "Hypertension increases the risk of stroke and heart attack."
text2 = "High blood pressure damages arterial walls over time, leading to cardiovascular events."
# Cosine Similarity Results:
# Our Model: 0.5941 (59.4%)
# NeuML Model: 0.5267 (52.7%)
# Improvement: +12.7%
Non-Medical Text Discrimination
Example 2: Medical vs. Programming Terms
texts = ["diabetes type 2", "asyncio.run()"]
# Cosine Similarity Results:
# Our Model: 0.0804 (8.0%) - Correctly identifies low similarity
# NeuML Model: 0.1926 (19.3%) - Higher false similarity
# Better Discrimination: 58% lower false positive rate
Key Advantages
- Enhanced Medical Understanding: 12.7% better similarity detection for related medical concepts
- Improved Discrimination: 58% reduction in false similarities between medical and non-medical terms
- Domain Specialization: Fine-tuned specifically on PubMed literature for optimal medical text processing
Training Details
- Optimizer: AdamW (learning rate: 3e-4, weight decay: 0.1)
- Batch Size: 72
- Training Steps: 7,000
- Warmup Steps: 700
Citation
If you use this model, please cite the base model paper and acknowledge this fine-tuning work.