Raw 500M Disjoint

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "l2t-project/raw-500m-disjoint"
)
tokenizer = AutoTokenizer.from_pretrained(
    "l2t-project/raw-500m-disjoint"
)

Citation

@article{yamaguchi2026enhancinglinguisticcompetencelanguage,
      title={Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks}, 
      author={Atsuki Yamaguchi and Maggie Mi and Nikolaos Aletras},
      year={2026},
      eprint={2601.03448},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.03448},
      journal={arXiv},
      volume={abs/2601.03448}
}