nanoGPT-3000
A 124M-parameter GPT-2-class causal language model built from scratch in PyTorch and trained on FineWeb-10B.
This is a pretraining checkpoint (3,000 steps, ~1.57B tokens).
Quick Stats
| Parameters | ~124M |
| Context | 2,048 tokens |
| Vocab | 50,304 (GPT-2 BPE, padded to 128) |
| Position enc | RoPE |
| Norm | RMSNorm + QK-Norm |
| Activation | Squared ReLU |
| Attention | FlashAttention-2 varlen (packed docs) |
| Skips | U-Net style |
| Optimizer | Muon (2D weights) + AdamW (embeddings) |
| Batch size | 524,288 tokens/step |
| Hardware | 2× A100 40GB |
| Time | ~70 min |
| MFU | 70% |
| Throughput | 375K tok/s |
WikiText-2
| Model | Tokens | PPL |
|---|---|---|
| OpenAI GPT-2 | ~40B | 25.2 |
| nanoGPT-3000 | ~1.57B | 40.50 |
How to Use
Repo (training code + inference): https://github.com/shehab-ashraf/nanogpt
Blog (deep dive): https://shehab-ashraf.github.io/posts/nanogpt/
Logs (W&B): https://wandb.ai/ashrafshehab-/nanoGPT