Bioformer-8L-RetroMAE

Bioformer-8L-RetroMAE is a compact biomedical BERT encoder (43M parameters) for dense retrieval. It was created by continuing to pretrain Bioformer-8L with RetroMAE, a pretraining objective designed for retrieval, on about 32 million PubMed titles and abstracts.

Important: This checkpoint is the RetroMAE-pretrained encoder. It has not been fine-tuned for retrieval. It is meant as a strong starting point (initialization) for training bi-encoder retrievers (for example, contrastive fine-tuning with hard negatives). Its raw [CLS] embeddings are not designed for zero-shot semantic search.

Model details

Why RetroMAE?

Standard masked language modeling (MLM) does not explicitly train the [CLS] vector, yet a bi-encoder uses that vector as the text representation. RetroMAE fixes this mismatch. The encoder reads a moderately masked input, and its [CLS] vector is passed to a lightweight one-layer decoder. The decoder must then reconstruct a separately and more heavily masked version of the same text from that single vector. This forces the [CLS] vector to summarize the whole text. The decoder is used only during pretraining, so the released encoder keeps the compact Bioformer-8L size and speed.

Training

Data

• PubMed records published through August 2026.
• Records without an abstract (title-only records) were excluded, as were records whose title plus abstract was shorter than 200 characters. About 32 million records remained.
• The title and abstract of each record were concatenated into a single training instance, with a maximum length of 512 tokens. Each [CLS] representation therefore corresponds to one PubMed record.

Procedure

• Initialization: the released bioformers/bioformer-8L checkpoint.
• Encoder masking: whole-word masking, probability 0.25.
• Decoder: single transformer layer; reconstruction masking probability 0.50. The decoder was discarded after pretraining.
• Loss: 0.5 × encoder MLM loss + 0.5 × RetroMAE decoder reconstruction loss.

How to use

Fine-tune as a bi-encoder retriever (recommended)

Use this checkpoint to initialize both the query encoder and the document encoder. Train with a contrastive objective such as InfoNCE, using in-domain query–document pairs and hard negatives. The setup we used in our paper:

• Separate query and document encoders, both initialized from this checkpoint.
• [CLS] pooling with L2 normalization; similarity is the inner product (equivalent to cosine similarity).
• InfoNCE loss, temperature τ = 0.05, with 1 positive and 100 BM25 hard negatives per question.
• 10 epochs, learning rate 3e-5, weight decay 0.01, linear decay, warmup ratio 0.10, gradient accumulation over 16 questions.
• Maximum length of 128 tokens for queries and 512 tokens for documents.

Evaluation

The results below are for bi-encoders fine-tuned from each backbone. All backbones were fine-tuned with the same BioASQ Task 12B training questions, the same BM25 hard negatives, and the same settings. They are therefore results of the downstream retriever, not of this checkpoint used zero-shot.

Retrieval used the full corpus of 28.8 million PubMed abstracts, evaluated on 340 questions from the four official BioASQ Task 12B golden test batches.

Intended uses and limitations

Intended uses

• Initializing dense retrievers or embedding models for biomedical literature search, including the retrieval component of RAG systems.
• Settings where encoding speed matters, such as indexing tens of millions of documents or encoding queries in real time. The encoder has the same size and inference speed as Bioformer-8L, which is about 3× faster than BERT-Base-sized models in CPU inference in our benchmark.

Limitations

• This checkpoint is not fine-tuned for retrieval; fine-tune it before use as a retriever.

Related models

• bioformers/bioformer-8L: base model, 8 layers, 43M parameters
• bioformers/bioformer-16L: deeper variant, 16 layers, 42M parameters
• Code: https://github.com/WGLab/bioformer

Contact

Li Fang (fangli9@mail.sysu.edu.cn) and Kai Wang (wangk@chop.edu)