GPT2-XL-Finetuned-WikiText103

Model Description

A fine-tuned version of GPT2-XL on the WikiText-103 dataset.

• Base Model: GPT2-XL
• Training Dataset: WikiText-103
• Model Size: 1.5B parameters

Performance on WikiText-103

Training Details

• Training Data: WikiText-103 (103M tokens)
• Optimizer: AdamW
• Learning Rate: 1e-5 with cosine schedule

Citation

This model was released as part of the paper "Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models" (NIPS'2025).

For more information, see: https://github.com/LUMIA-Group/MemoryDecoder.

If you use this model, please cite:

@article{cao2025memory,
  title={Memory decoder: A pretrained, plug-and-play memory for large language models},
  author={Cao, Jiaqi and Wang, Jiarui and Wei, Rubin and Guo, Qipeng and Chen, Kai and Zhou, Bowen and Lin, Zhouhan},
  journal={arXiv preprint arXiv:2508.09874},
  year={2025}
}