GPT-2 (124M) trained from scratch on FineWeb-Edu

A GPT-2 small (124M parameters) language model that I trained from scratch, following Andrej Karpathy's "Let's reproduce GPT-2 (124M)" video. This is a base model: it only predicts the next token. It is not instruction-tuned and has no safety tuning.

Results (final checkpoint, step 56383)

The comparison with OpenAI's GPT-2 is not like-for-like: this model was trained on FineWeb-Edu (educational web text), OpenAI's on a different data mix. One standard error on HellaSwag at this accuracy is about +/- 0.47 points.

Evaluation (zero-shot, lm-evaluation-harness)

Both models were run through the same EleutherAI lm-evaluation-harness code, zero-shot, in float32. Accuracies are in % with one standard error (±); perplexities are lower-is-better.

Bold = the better of the two on that row. Many differences are within one or two standard errors so read the ± before claiming a win.

Notes:

• The HellaSwag number in the Results section above (32.84%) comes from my training-time evaluation, which normalises by token count; the harness's acc_norm normalises by byte length, so it reads higher for both models. Compare each model with OpenAI's under the same method, not across methods.
• This model is behind on LAMBADA (fiction-style last-word prediction). FineWeb-Edu is filtered toward educational text and contains little narrative writing, which is a plausible reason; I have not run an ablation to confirm it.
• BoolQ and COPA are near chance for both models at this size; treat them as sanity checks, not as signal.
• Evaluated 2026-11-09.

Training

• Architecture: GPT-2 small (12 layers, 12 heads, 768 width, 1024 context), 124M parameters; same vocabulary as GPT-2 (50,257)
• Data: FineWeb-Edu 10B-token sample (99 train shards, 9.85B tokens), 3 epochs = 29.6B tokens seen
• Batch: 524,288 tokens per step, 56,384 steps
• Optimizer: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 6e-4, 715 warmup steps, cosine decay to 6e-5, grad clip 1.0
• Precision: bf16 autocast, flash attention; DDP on 4 rented GPUs (Vast.ai); about 49 hours
• Training and evaluation code: based on Karpathy's nanoGPT / build-nanogpt, with small changes (HellaSwag from the HF Hub, checkpoints pushed to the Hub, TensorBoard logging)

The runs/ folder holds the TensorBoard logs of the whole run (see the Training metrics tab) and log.txt the raw loss log.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "roger33303/gpt2-124m-fineweb-edu"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Hello, I'm a language model,", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True, top_k=50, temperature=0.8,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0]))

Limitations

• Small base model: it can write fluent-looking text but often makes factual errors and repeats itself.
• Not instruction-tuned, not aligned, no safety filtering. It can produce wrong or inappropriate text.
• Trained on English web text only, 1024-token context.
• Converted from my training checkpoint (model_56383.pt) to the Hugging Face GPT-2 format; the converted model matches the original's logits (max absolute difference below 1e-3 on random tokens).