GPT-2 124M trained from scratch on FineWeb-Edu
A GPT-2 small (124M parameters) trained from random initialisation on 9.9B tokens of FineWeb-Edu, using my own training code: github.com/AnjaneyaluAJ/llm-pretraining. It scores 30.59% on HellaSwag, above OpenAI's GPT-2 124M (29.58%).
Use it
from transformers import pipeline
generator = pipeline("text-generation", model="Anjaneyalu/gpt2-124m-fineweb-edu")
print(generator("Photosynthesis is the process", max_new_tokens=40, do_sample=False)[0]["generated_text"])
# Photosynthesis is the process by which plants convert sunlight into energy. ...
It is a standard GPT2LMHeadModel with the GPT-2 tokenizer, so it works anywhere GPT-2 does.
Results
Both models were evaluated with the same code on the same data.
| OpenAI GPT-2 124M | This model | |
|---|---|---|
| HellaSwag acc_norm (0-shot, 10,042 val examples) | 29.58% | 30.59% |
| HellaSwag acc | 28.56% | 29.14% |
| FineWeb-Edu val loss | 3.262 | 2.990 |
| FineWeb-Edu val perplexity | 26.10 | 19.89 |
- HellaSwag is the neutral comparison. This model passes OpenAI's score at about 6.8B training tokens. Our evaluation gives OpenAI's GPT-2 29.58%, matching the 29.55% reported in karpathy/build-nanogpt.
- The FineWeb-Edu numbers favour this model, because they are measured on held-out text from its own training distribution.
Training
| Architecture | GPT-2 124M: 12 layers, 768 hidden, 12 heads, context 1024, tied embeddings |
| Data | FineWeb-Edu sample-10BT, one epoch: 9,944,035,344 tokens |
| Batch | 524,288 tokens per step, 18,966 steps |
| Optimizer | AdamW, peak lr 6e-4, 700 warmup steps, cosine decay to 6e-5, weight decay 0.1, grad clip 1.0 |
| Precision | bf16 mixed precision, torch.compile |
| Compute | 4x NVIDIA A10G 24 GB, 16.4 hours (~177k tokens/s, ~54% MFU) |
Full config, metrics, logs and charts:
results/gpt2_124m_fineweb.
The weights were converted from the training checkpoint with export_hf.py from the same repo;
the converted model reproduces the original logits to within 3e-5.
Limitations
This is a small base model trained for research and learning, not an assistant. It has not been instruction-tuned or safety-tuned. It continues text rather than following instructions, it often repeats itself, and it states false things fluently. Its knowledge is limited to English educational web text. Don't use it for anything where accuracy matters.
Credits
- The model and training code build on Sebastian Raschka's LLMs-from-scratch and his book Build a Large Language Model (From Scratch) (Manning, 2024).
- The training recipe and the HellaSwag comparison follow Andrej Karpathy's build-nanogpt.
- Data: FineWeb-Edu by Hugging Face (ODC-By 1.0).
- Benchmark: HellaSwag (Zellers et al., 2019).
Author: Anjaneyalu Thippaiah