mmBERT-small (Safetensors Format)

This repository provides an unofficial Safetensors-converted version of jhu-clsp/mmBERT-small. Converting the weights to Safetensors improves loading security and allows you to use the model with recent versions of the transformers library without requiring trust_remote_code=True.

Overview

mmBERT-small is a state-of-the-art multilingual encoder trained on 3T+ tokens across 1,800+ languages, utilizing novel techniques for learning low-resource languages during the annealing/decay phase.

Usage

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("agentlans/mmBERT-small-safetensors")
model = AutoModel.from_pretrained("agentlans/mmBERT-small-safetensors")

Citation

If you use mmBERT in your research, please cite the original authors:

@misc{marone2025mmbertmodernmultilingualencoder,
      title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, 
      author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2509.06888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.06888}, 
}