GPT-2 trained on C4 — Lab 4, AD

This is the lab4-yoru-AD-488712 checkpoint, trained from scratch on English C4. It has the lowest local validation loss of our runs so far. The reported score is a local evaluation, not an official online-judge result.

Model

• Stock Hugging Face GPT2LMHeadModel: 12 layers, 12 attention heads, hidden width 768, context length 1,024, vocabulary 50,257, tied input/output embeddings.
• 124,439,808 unique parameters, saved in FP32.
• GPT-2 tokenizer. No pretrained model weights were used.

Training

• Data: C4 en train, tokenized with the GPT-2 tokenizer without BOS/EOS. Documents were truncated to 1,024 tokens and best-fit packed into 1,024-token rows, with positions restarting at 0 for each document.
• Attention was restricted to each document during training (varlen flash attention). This is a training-time kernel only; the saved model is plain GPT-2.
• Exact deduplication. The first 100,000 documents of validation.shuffle(seed=42) (the lab's test set) were excluded.
• AdamW (fused): learning rate 1.8e-3, betas (0.9, 0.95), eps 1e-8, weight decay 0.1 on matrices only, gradient clipping 1.0, dropout 0, random seed 42.
• Schedule: 150 warmup steps, full learning rate for the first half of the training time, then linear decay to 0.
• Batch: 256 rows × 1,024 tokens = 262,144 token slots per step (including padding). 6,010 steps, about 1.58B token slots.
• The vocabulary was padded to 50,304 during training and sliced back to 50,257 before saving.
• Two NVIDIA H200 GPUs, BF16 autocast, torch.compile, 29 minutes of training. PyTorch 2.14.0, Transformers 5.17.0.

Evaluation

The evaluation set is the 10,000 C4 en validation documents that follow the lab's 100,000 test documents in validation.shuffle(seed=42). It is scored like the lab's judge:

• Each document is scored on its own, from position 0, with no BOS.
• Each document is truncated to 1,024 tokens.
• The score is total loss divided by the number of scored tokens.

Load the checkpoint

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yorukot/gpt2-traning"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

This is a base language model for an educational experiment, without instruction tuning. Its web training data can produce inaccurate or biased text.

Training run on W&B.