This is a GPT-2 (1558M) model trained in llm.c for 100B tokens with cosine LR on Fineweb-Edu.
A lot more detailed info and observations are here: https://x.com/Yuchenj_UW/status/1814703583453192272
This is a GPT-2 (1558M) model trained in llm.c for 100B tokens with cosine LR on Fineweb-Edu.
A lot more detailed info and observations are here: https://x.com/Yuchenj_UW/status/1814703583453192272
Have the files? Submit a torrent.
Submit a torrent →Have the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo yuchenj/gpt2_1558M_100B_FinewebEdu_hf ./model-folder