BiomedBERT fine-tuned on MedMentions (Biologic Function subtypes)

This model is a named entity recognition (NER) model for biomedical text. It finds mentions of biologic functions and diseases in text. It also gives each mention one of nine UMLS semantic types.

The backbone is microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract. We fine-tuned the backbone on the MedMentions corpus.

Entity types

All nine labels are UMLS semantic types under Biologic Function (T038).

Label	UMLS semantic type	TUI
`PHYSIOL_FUNC`	Physiologic Function	T039
`ORGANISM_FUNC`	Organism Function	T040
`ORGAN_TISSUE_FUNC`	Organ or Tissue Function	T042
`CELL_FUNC`	Cell Function	T043
`MOL_FUNC`	Molecular Function	T044
`PATHOL_FUNC`	Pathologic Function	T046
`DISEASE_SYNDROME`	Disease or Syndrome	T047
`CELL_MOLECULE_DYSFUNC`	Cell or Molecular Dysfunction	T049
`EXP_DISEASE_MODEL`	Experimental Model of Disease	T050

How to use

Install the libraries:

pip install transformers torch

Run the model with the pipeline API:

from transformers import pipeline
 
ner = pipeline(
    "token-classification",
    model="HowML/BiomedBERT-ner-medmentions-fine-tuned", 
    aggregation_strategy="simple",  # merge sub-word tokens into full mentions
)

Evaluation

Overall results (test set, entity level, seqeval)

The scores below are strict entity-level scores. A prediction is correct only if the span boundaries and the type are both correct.

Entity type	Precision	Recall	F1	Support
`CELL_FUNC`	0.49	0.60	0.54	506
`CELL_MOLECULE_DYSFUNC`	0.52	0.52	0.52	105
`DISEASE_SYNDROME`	0.75	0.78	0.76	3,731
`EXP_DISEASE_MODEL`	0.29	0.36	0.32	45
`MOL_FUNC`	0.59	0.64	0.61	1,463
`ORGANISM_FUNC`	0.49	0.46	0.48	1,343
`ORGAN_TISSUE_FUNC`	0.42	0.32	0.37	296
`PATHOL_FUNC`	0.53	0.54	0.54	689
`PHYSIOL_FUNC`	0.30	0.17	0.22	208
Micro avg	0.62	0.64	0.63	8,386
Macro avg	0.49	0.49	0.48	8,386
Weighted avg	0.62	0.64	0.63	8,386

CELL_FUNC (possible = 506, actual = 611)

Mode	Correct	Incorrect	Partial	Missed	Spurious	Precision	Recall	F1
strict	302	64	0	140	245	0.494	0.597	0.541
exact	302	64	0	140	245	0.494	0.597	0.541
partial	302	0	64	140	245	0.547	0.660	0.598
ent_type	366	0	0	140	245	0.599	0.723	0.655

DISEASE_SYNDROME (possible = 3,731, actual = 3,854)

Mode	Correct	Incorrect	Partial	Missed	Spurious	Precision	Recall	F1
strict	2,894	260	0	577	700	0.751	0.776	0.763
exact	2,894	260	0	577	700	0.751	0.776	0.763
partial	2,894	0	260	577	700	0.785	0.811	0.797
ent_type	3,154	0	0	577	700	0.818	0.845	0.832