CuriosLM/gpt2-124m-fineweb-edu

Model A: GPT-2 small (124M) trained from scratch, the baseline of a controlled comparison with a Gemma-style model trained identically.

A small research model from the CuriosLM project, released for reproducibility and study. It is a base model: not instruction-tuned and not safety-tuned, so it can produce false, biased or nonsensical text.

Read more in the write-up, Part 1, pretraining and Part 2, inference, positions, attention and context extension; the code is at Shinibali/CuriosLM.

Architecture

GPT-2 small, unchanged: 12 layers, 12 heads, d_model 768, learned absolute positions (1,024), GELU MLP, pre-LayerNorm, biases on.

Training

Data: 100% HuggingFaceFW/fineweb-edu (sample-10BT)
Tokens: 9.83B, 20,000 optimizer steps
AdamW, peak learning rate 0.0006 with 760 warmup steps and cosine decay to 6e-05, weight decay 0.1, seed 1337
Tokens per step: 491,520

Results

Evaluation	Score
Perplexity, held-out FineWeb-Edu	21.89
Perplexity, held-out Cosmopedia-v2	20.31
ARC-Easy (acc)	52.4%
HellaSwag (acc_norm)	32.4%
LAMBADA (acc)	24.2%
LAMBADA (perplexity)	111.1

Perplexity on about 2M held-out tokens of each dataset; benchmarks zero-shot with EleutherAI's lm-evaluation-harness.

Usage

Standard 🤗 GPT-2 format:

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CuriosLM/gpt2-124m-fineweb-edu")
model = AutoModelForCausalLM.from_pretrained("CuriosLM/gpt2-124m-fineweb-edu")
ids = tokenizer("The mitochondria is", return_tensors="pt").input_ids
print(tokenizer.decode(model.generate(ids, max_new_tokens=40, do_sample=True, top_k=50)[0]))

Loading…