Vitiligo Relation Extractor

A biomedical relation extraction model built for the Vitiligo Initiative, an open, non-profit research platform. Given a passage whose chemicals, diseases, genes and variants are already annotated, it classifies the relation between every pair of them into BioRED's eight types. It reads the text PubTator 3.0 does not: full-text paper bodies and abstracts outside PubMed.

Model details

• Architecture: BiomedBERT encoder with ATLOP localized context pooling (joint pair self-attention), closest-mention between-span pooling, token- distance / co-sentence embeddings, and entity-type domain masking. A passage is encoded once and every entity pair in it is scored from that single pass.
• Base model: BiomedBERT (base, uncased, PubMed abstracts and full text)
• Labels: Association, Positive_Correlation, Negative_Correlation, Bind, Cotreatment, Comparison, Drug_Interaction, Conversion, or none
• Entity types: Chemical, Disease, Gene, Variant
• Parameters: 113M · Language: English · License: Apache-2.0
• Code: vitiligo.relations

Uses

Intended: reading relations out of biomedical literature to build a knowledge graph, where each relation keeps the sentence it came from so it can be checked. In the Vitiligo Initiative, its relations become graph edges with lower confidence than curated or registry sources.

Out of scope: clinical decisions, or any use where an unchecked relation is taken as fact. It does not find entities itself; they must be annotated first, for example by PubTator 3.0.

How to use

The model runs through the project's own code, not a transformers class.

pip install "vitiligo[train] @ git+https://github.com/recepsirin/vitiligo-initiative"

from vitiligo.relations.train import load

model, bias, settings = load("re-cep/vitiligo-relation-extractor")

Append @<revision> to the id to pin one version. To reproduce the BioRED test scores:

vitiligo relations evaluate --model re-cep/vitiligo-relation-extractor --split Test

Training

1. Silver: PubTator 3.0's relations in 11,106 abstracts of the Vitiligo Initiative corpus, excluding every BioRED abstract and a fixed 1-in-20 held-out set.
2. Gold: BioRED's 400 training abstracts, starting from the silver model.

The epoch (5 of 10) and the decision threshold (bias = 0.4) were chosen on BioRED's development split (F1 0.603). Per-epoch development scores are in relations.json.

Evaluation

BioRED test split, relations between given entities, micro-averaged as in the BioRED paper. Pair + type precision is 0.624 and recall 0.580 (pair-only precision is 0.758 and recall is 0.704).

The BioRED paper's PubMedBERT reaches 0.729 and 0.589 by encoding each pair separately; this model surpasses both (0.730 and 0.602) while encoding each passage only once, which is what makes reading 821,413 candidate pairs in full texts practical.

In use on full texts. Of 120 relations read from full-text bodies and checked by hand against their sentences, 68% were correct. The Vitiligo Initiative therefore loads only relations with probability ≥ 0.8, outside tables and figures, excluding co-administration claims, and accepting "treats" only when the disease is vitiligo. On a fresh hand-checked sample under that rule, 45 of 60 relations (75%) were correct, and 15 of 17 where vitiligo is one of the pair.

Limitations

• About a quarter of the relations it loads in practice are wrong; each should be read with the sentence it came from.
• Table rows, figure captions and lists of drugs are frequent sources of error.
• The silver training data come from one disease's literature; accuracy on other domains is untested beyond BioRED.

Files

License and attribution

Released under Apache-2.0. Fine-tuned from BiomedBERT (© Microsoft, MIT License). Trained on BioRED, distributed by NCBI as freely available, and on PubTator 3.0 annotations, a work of the US Government.

Citation

• Luo L, Lai PT, Wei CH, Arighi CN, Lu Z. BioRED: a rich biomedical relation extraction dataset. Briefings in Bioinformatics 2022;23(5):bbac282.
• Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 2021;3(1).
• Wei CH, Allot A, Lai PT, et al. PubTator 3.0: an AI-powered literature resource for unlocking biomedical knowledge. Nucleic Acids Research 2024;52(W1):W540-W546.
• Zhou W, Huang K, Ma T, Huang J. Document-level relation extraction with adaptive thresholding and localized context pooling. AAAI 2021.