Loading…
Loading…
A GPT-2 small (124M parameters) trained from random initialisation on 9.9B tokens of FineWeb-Edu, using my own training code: github.com/AnjaneyaluAJ/llm-pretraining. It scores 30.59% on HellaSwag, above OpenAI's GPT-2 124M (29.58%).
from transformers import pipeline
generator = pipeline("text-generation", model="Anjaneyalu/gpt2-124m-fineweb-edu")
print(generator("Photosynthesis is the process", max_new_tokens=40, do_sample=False)[0]["generated_text"])
# Photosynthesis is the process by which plants convert sunlight into energy. ...
It is a standard GPT2LMHeadModel with the GPT-2 tokenizer, so it works anywhere GPT-2 does.
Both models were evaluated with the same code on the same data.
| OpenAI GPT-2 124M | This model | |
|---|---|---|
| HellaSwag acc_norm (0-shot, 10,042 val examples) | 29.58% | 30.59% |
| HellaSwag acc | 28.56% | 29.14% |
| FineWeb-Edu val loss | 3.262 | 2.990 |
| FineWeb-Edu val perplexity | 26.10 | 19.89 |
| Architecture | GPT-2 124M: 12 layers, 768 hidden, 12 heads, context 1024, tied embeddings |
| Data | FineWeb-Edu sample-10BT, one epoch: 9,944,035,344 tokens |
| Batch | 524,288 tokens per step, 18,966 steps |
| Optimizer | AdamW, peak lr 6e-4, 700 warmup steps, cosine decay to 6e-5, weight decay 0.1, grad clip 1.0 |
| Precision | bf16 mixed precision, torch.compile |
| Compute | 4x NVIDIA A10G 24 GB, 16.4 hours (~177k tokens/s, ~54% MFU) |
Full config, metrics, logs and charts:
results/gpt2_124m_fineweb.
The weights were converted from the training checkpoint with export_hf.py from the same repo;
the converted model reproduces the original logits to within 3e-5.
This is a small base model trained for research and learning, not an assistant. It has not been instruction-tuned or safety-tuned. It continues text rather than following instructions, it often repeats itself, and it states false things fluently. Its knowledge is limited to English educational web text. Don't use it for anything where accuracy matters.
Author: Anjaneyalu Thippaiah