BiomedBERT Small

This is a 22.7M parameter BERT encoder-only model trained on data from PubMed. The raw data was transformed using PaperETL with the results stored as a local dataset via the Hugging Face Datasets library.

This model is designed to be a solid-performing small model fitting in between the 110M parameter BiomedBERT Base model and the tiny BiomedBERT Hash series of models.

Usage

biomedbert-small can be loaded using Hugging Face Transformers as follows.

from transformers import AutoModel

model = AutoModel.from_pretrained("neuml/biomedbert-small")

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

Evaluation Results

This Medical Abstracts Text Classification Dataset was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.

Metrics were generated using Hugging Face's standard run_glue script as shown below.

python run_glue.py --model_name_or_path neuml/biomedbert-small --dataset-name medclassify --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 1e-4 --num_train_epochs 4 --output_dir outputs --trust-remote-code True

Note: The original dataset was saved locally as medclassify the the condition_label column renamed to label to work more easily with the glue script

As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!

More Information

Read more about the model in this article.