AntiBERTy

AntiBERTy is an antibody-specific transformer language model pre-trained on 558M natural antibody sequences from the Observed Antibody Space, as described in Deciphering antibody affinity maturation with language models and weakly supervised learning. It is an 8-layer BERT encoder (512 hidden units, 8 attention heads, 26M parameters) trained with masked language modelling together with species and chain-type classification heads, and provides the sequence features used by IgFold for antibody structure prediction.

Files

Usage

Install the antiberty package; the weights are downloaded from this repository on first use and cached.

pip install antiberty
antiberty classify EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS

from antiberty import AntiBERTyRunner

antiberty = AntiBERTyRunner()
embeddings = antiberty.embed(["EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS"])
species, chains = antiberty.classify([...])
filled = antiberty.fill_masks(["QVQLQESGGGLVQAGGSLTLSCAVSG__FSNYAMG"])

The model class (antiberty.AntiBERTy, a transformers.BertPreTrainedModel with the extra classification heads) and the tokenizer live in the package; the weights can also be loaded with AntiBERTy.from_pretrained(<local snapshot>, attn_implementation="eager").

Intended use and limitations

AntiBERTy expects antibody variable-domain sequences (roughly 100-130 residues per chain, maximum 510). It was trained on heavy and light chains from human, mouse, rat, rabbit, rhesus and camelid repertoires; outputs for other proteins or full-length antibody chains are not meaningful.

Citation

@article{ruffolo2021deciphering,
    title = {Deciphering antibody affinity maturation with language models and weakly supervised learning},
    author = {Ruffolo, Jeffrey A and Gray, Jeffrey J and Sulam, Jeremias},
    journal = {arXiv preprint arXiv:2112.07782},
    year= {2021}
}