nanoprot-esm2-S

A protein language model (esm2 architecture) — 33.3M (33,263,040) parameters, trained on 399.2M (399,156,480) UniRef50 residues.

nanoprot-esm2-S is part of the nanoprot suite: a Pythia-style matrix of protein language models spanning three architectures (gpt2, esm2, mamba) and four scales (XS/S/M/L), each trained from scratch on UniRef50 under a matched, Chinchilla-style data budget. The suite is built for controlled comparison — same data, same tokenizer, one variable at a time.

Headline result

Validation masked cross-entropy (bits/residue over masked positions): 3.6336 ± 0.0029 (n=3 seeds)

This is a masked-LM pseudo-perplexity over masked positions — a different quantity from the autoregressive models' bits-per-residue. Compare esm2 models to each other and on downstream / probing tasks, not via this number against gpt2/mamba.

Model details

Training

Evaluation

Evaluated on a held-out UniRef50 shard. Validation masked cross-entropy (bits/residue over masked positions): 3.6336 ± 0.0029 (n=3 seeds).

This is a masked-LM pseudo-perplexity over masked positions — a different quantity from the autoregressive models' bits-per-residue. Compare esm2 models to each other and on downstream / probing tasks, not via this number against gpt2/mamba.

Intended use & limitations

Research use: learning protein representations, extracting residual-stream features, mechanistic-interpretability probing, and architecture comparison. Trained only on UniRef50 sequences — not for clinical or diagnostic use, and not aligned to any downstream task out of the box.

How to load

Install nanoprot (pip install nanoprot), download this repo, and point the arch-aware loader at the folder — it works for any nanoprot architecture (gpt2 / esm2 / mamba), reading the embedded config and selecting the right tokenizer automatically.

from nanoprot.training.checkpoint import load_pretrained

model, cfg, meta, tokenizer = load_pretrained(
    "path/to/this/repo", device="cpu", return_tokenizer=True,
)
model.eval()
# meta carries the trained-artifact facts (params, FLOPs, val metric, ...)

The nanoprot suite

Hub: yagizdevre/nanoprot-esm2-S (seed 0 is the default; siblings on branches seed1, seed2). This model's sibling seeds:

• nanoprot-esm2-S-s0 — final masked-cross-entropy-bits 3.6332
• nanoprot-esm2-S-s1 — final masked-cross-entropy-bits 3.6310
• nanoprot-esm2-S-s2 — final masked-cross-entropy-bits 3.6367

The full suite spans {gpt2, esm2, mamba} x {XS, S, M, L} x {seed 0,1,2}. See the nanoprot repository for the complete grid and the scaling-curve comparisons.

Citation

@software{nanoprot,
  author  = {Devre, H. Yagiz},
  title   = {nanoprot: a minimal training framework for protein language models},
  year    = {2026},
  url     = {https://github.com/ygzdvr/nanoprot}
}

Reproducibility

Trained with nanoprot v0.5.0 (corpus prepared 2026-06-01T12:51:52+00:00). The exact, complete training config is in config.yaml (also embedded in meta_000761.json). Re-train with:

python -m scripts.train --config config.yaml