Antibody language models (AbLMs) help decode immune repertoires for therapeutic discovery, but optimal scaling strategies remain unclear. We trained ESM-2-based AbLMs across five sizes (8M–650M parameters) and three paired Ab sequences dataset scales (~1.6M total sequences). Results are described in our preprint on bioRxiv. Effective AbLM scaling requires balancing model size with data availability to avoid diminishing returns. The pre-trained models and the datasets used to train them are available on Zenodo and code is available on GitHub.
All 15 models of the study are in the AbLMs Scaling Laws collection.
Load the model and tokenizer as follows:
from transformers import EsmTokenizer, EsmForMaskedLM
model = EsmForMaskedLM.from_pretrained("brineylab/150M_quarter_checkpoint-165000")
tokenizer = EsmTokenizer.from_pretrained("brineylab/150M_quarter_checkpoint-165000") # the standard ESM-2 tokenizer
The models were trained on natively paired antibody sequences. The input is the heavy-chain amino acid sequence, two <cls> tokens as a separator, and the light-chain amino acid sequence (during training, inputs were padded or truncated to 320 tokens):
heavy = "EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLEWVARIYPTNGYTRYADSVKGRFTISADTSKNTAYLQMNSLRAEDTAVYYCSRWGGDGFYAMDYWGQGTLVTVSS"
light = "DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKLLIYSASFLYSGVPSRFSGSRSGTDFTLTISSLQPEDFATYYCQQHYTTPPTFGQGTKVEIK"
inputs = tokenizer(heavy + "<cls><cls>" + light, return_tensors="pt")
outputs = model(**inputs)
The model can be finetuned for classification tasks (such as specificity and pair classification in the paper) by loading the model with a sequence classification head:
from transformers import EsmForSequenceClassification
model = EsmForSequenceClassification.from_pretrained("brineylab/150M_quarter_checkpoint-165000")
# freeze the base model weights prior to finetuning
for param in model.base_model.parameters():
param.requires_grad = False