Ettin Checkpoints

This repository contains the raw training checkpoints for the Ettin models. Each model contains a unique subdirectory, e.g. enc-150m for Ettin-Encoder-150m, with three subfolders for decay, ext, and pretrain.

These files work with Composer and contain all state needed to resume pre-training. Please see the ModernBERT repository for usage details.

🔗 Related Resources

• Models: Ettin Model Suite (17M-1B parameters)
• Phase 1: Pre-training Data (1.7T tokens)
• Phase 2: Mid-training Data (250B tokens)
• Phase 3: Decay Phase Data (50B tokens)
• Training Order: Batch-level Data Order
• Paper: Arxiv link
• Code: GitHub Repository

Citation

@misc{weller2025seqvsseqopen,
      title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders}, 
      author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2507.11412},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11412}, 
}