gpt2-30m-fineweb-edu

A ~30M-parameter GPT-2 style decoder pretrained from scratch (random init) on HuggingFaceFW/fineweb-edu (sample-10BT config, streaming) in a single ~38-minute run on one NVIDIA A10G as a budget demo of the full pretraining pipeline.

Architecture

6 layers, hidden 384, 6 heads, vocab 50257 (GPT-2 BPE tokenizer), context 512
tied input/output embeddings, ~30.1M parameters

Training

CLM loss on packed 512-token blocks, bf16
AdamW β=(0.9, 0.95), weight decay 0.1, grad clip 1.0
lr 6e-4, cosine decay, 150-step warmup, effective batch 64×512 = 32,768 tokens/step
4,000 optimizer steps ≈ 131M tokens seen (1 epoch over the streamed subset)

Results

Metric	Value
final eval loss	4.7510
eval perplexity	≈ 115.7 (exp of eval loss)
train loss (end)	~4.9–5.0

The model is heavily undertrained (131M tokens for 30M params) — expect repetitive, low-coherence text. It demonstrates the pipeline, not quality.

Usage

from transformers import pipeline
pipe = pipeline("text-generation", "zhouaz/gpt2-30m-fineweb-edu")
print(pipe("The history of the Roman Empire begins", max_new_tokens=80, do_sample=True)[0])

Reproduction

Training script pattern follows transformers examples/pytorch/language-modeling/run_clm.py: streaming dataset → online tokenization → block packing → Trainer with max_steps, periodic push_to_hub (every 500 steps).

Loading…