MiniGPT - TinyStories

A ~5M parameter character-level GPT model trained from scratch on TinyStories.

Model Details

Architecture

Standard mini-GPT architecture:

• Token Embedding + Position Embedding
• 6 Transformer Blocks (Causal Self-Attention + MLP)
• LayerNorm + Residual Connections
• Weight Tying (input embedding = output projection)

Training

Trained on CPU for ~1 hour 17 minutes on the TinyStories dataset.

• Optimizer: AdamW (lr=3e-4, weight_decay=0.01)
• Schedule: Warmup (200 steps) + Cosine Decay
• Batch Size: 64, Block Size: 128

Usage

import torch
from train_torch import MiniGPT

# Load model
model = MiniGPT(vocab_size=72, n_embd=256, n_head=8, n_layer=6, block_size=128, dropout=0.1)
model.load_state_dict(torch.load("model.safetensors"))
model.eval()

# Generate
context = torch.zeros((1, 1), dtype=torch.long)
output = model.generate(context, max_new_tokens=200, temperature=0.7)

Sample Generation (Temperature=0.7)

He saw a bike with a big box. He was a girl named Sue. Sue liked to give it very happy. One day, she got to the little boy named Tim. Tim was very happy. He wind the best friends was friends. The tree...

License

MIT