Quick start guide

To use this models, follow the snippet below:

from transformers import AutoModelForMaskedLM

# model_config_overrides = {}  # Use this to optionally override config parameters
model = AutoModelForMaskedLM.from_pretrained(
    "kuleshov-group/proseco-owt",
    trust_remote_code=True,
    # **model_config_overrides,
)

Model details

• Trained from scratch on Skylion007/openwebtext
• GPT2-tokenizer

See the paper for full details

Citation

@article{
  schiff2026learn,
  title={Learn from Your Mistakes: Self-Correcting Masked Diffusion Models},
  author={Schiff, Yair and Belhasin, Omer and Uziel, Roy and Wang, Guanghan and Arriola, Marianne and Turok, Gilad and Elad, Michael and Kuleshov, Volodymyr},
  journal={arXiv preprint arXiv:2602.11590},
  year={2026}
}