alberta-uspto-2800k

ALBERT model pretrained from scratch by masked language modeling (MLM) on ~2.8M USPTO reactions, for attention-guided atom mapping with RXNMapper v2. The atom-mapping signal is learned unsupervised, it emerges in one attention head of the model.

Intended use

These models were pretrained purely by masked language modeling on unlabeled reaction SMILES, with no task-specific supervision. Any checkpoint can therefore be used freely for any downstream purpose: atom mapping, fine-tuning, or feature extraction.

What's in this repo

• Recommended checkpoint at the repo root (config.json + model.safetensors): this is checkpoint-1360000, which performs very well. Use it with head 3, layer 10.
• All 150 training checkpoints (steps 10,000 → 1,500,000, every 10k) in checkpoint-<step>/ subfolders, so you can find the checkpoint that works best on your own validation set.
• Shared tokenizer (vocab.txt: 591 tokens, special_tokens_map.json, tokenizer_config.json) at the repo root; the vocabulary is identical across checkpoints.

Usage

Download the repo and point the RXNMapper v2 tooling at it:

hf download yvsgrndjn/alberta-uspto-2800k --local-dir alberta-uspto-2800k

The downloaded folder already matches the layout expected by the RXNMapper v2 scripts: the checkpoint-<step>/ subfolders are picked up directly by scripts/evaluate_models.py (to compare checkpoints on a validation set) and by the mapping scripts. Pass the checkpoint folder as model_path and the root vocab.txt as vocab_path. See the project README for full instructions.

Citation

If you use this work, please cite our work:

@article{grandjeanRXNMapperv2UpdatedValidation2026
      title={RXNMapperv2: updated validation and evaluation on expanded chemical space}, 
      author={Grandjean, Yves and Raffaud, Sacha, and Westerlund, Annie M. and Schwaller, Philippe and Genheden, Samuel and Reymond, Jean-Louis},
      journal={ChemRxiv}, 
      year={2026},
      doi={10.26434/chemrxiv.15005247/v1}
      URL={https://chemrxiv.org/doi/full/10.26434/chemrxiv.15005247/v1},
      note= {Preprint}
}

and the original RXNMapper paper:

@article{schwaller2021extraction,
  title={Extraction of organic chemistry grammar from unsupervised learning of chemical reactions},
  author={Schwaller, Philippe and Hoover, Benjamin and Reymond, Jean-Louis and Strobelt, Hendrik and Laino, Teodoro},
  journal={Science Advances},
  volume={7},
  number={15},
  pages={eabe4166},
  year={2021},
  publisher={American Association for the Advancement of Science}
}