Mini-GPT TinyStories

A compact GPT-2-style language model (~17.7M parameters) trained from scratch on the TinyStories dataset — a collection of 2.1M synthetically generated short stories designed for small-scale language model training.

Model Details

Evaluation Results

A perplexity of 4.81 is excellent for TinyStories — the model has learned the simple vocabulary and story structure well. For comparison, random prediction would yield perplexity ~50,257 (vocab size).

Usage

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "bhautikv/mini-gpt-tinystories"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
model.to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "Once upon a time, a little girl named Lily"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output, skip_special_tokens=True))

Batch Generation

prompts = [
    "Once upon a time, a little girl named Lily",
    "The brave knight approached the dragon and",
    "One day, a small bird found a shiny",
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    print(tokenizer.decode(output, skip_special_tokens=True))
    print("---")

Training Procedure

Training Hyperparameters

Training Results

Training Infrastructure

• Platform: Kaggle Notebook (GPU T4 x2)
• GPUs: 2× NVIDIA T4 (16GB each)
• Distributed strategy: Accelerate launch with DistributedDataParallel
• Experiment tracking: Trackio (Hugging Face)
• Training time: ~6 hours (including data preprocessing)

Data Preprocessing

1. Loaded 2,119,719 training stories and 21,990 validation stories from TinyStories
2. Tokenized using GPT-2 BPE tokenizer (vocab size 50,257)
3. Concatenated all tokens and grouped into fixed 256-token blocks
4. Used DataCollatorForLanguageModeling with dynamic padding for training

Intended Uses & Limitations

Intended Uses

• Educational demonstration of small-scale language model training
• Generating simple children's stories with coherent structure
• Baseline for experimenting with GPT-2 architecture modifications
• Learning resource for distributed training with Hugging Face Accelerate

Limitations

• Only trained on TinyStories — limited to simple vocabulary and short narratives
• Small model size (17.7M params) limits complexity of generated stories
• Context length of 256 tokens restricts long-form generation
• Not suitable for production use, real-world text generation, or handling complex topics
• May produce repetitive or nonsensical text outside the TinyStories domain

Framework Versions

• Transformers 5.14.1
• Pytorch 2.10.0+cu128
• Datasets 5.0.0
• Tokenizers 0.22.2
• Accelerate (latest)
• Trackio (latest)