MicroGPT - Tinystories

A minimal character-level GPT model trained from scratch on the tinystories dataset.

Model Details

Quick Start

# Clone the repo
git clone https://huggingface.co/{{cookiecutter.repo_id if cookiecutter else 'your-username/micro-gpt-' + config['dataset']}}
cd micro-gpt-tinystories

# Install dependencies
pip install -r requirements.txt

# Generate text
python inference.py --prompt "Once upon a time"

Usage in Python

import torch
from models.micro_gpt import MicroGPT

# Load config
import json
with open('config.json') as f:
    config = json.load(f)

# Build model
model = MicroGPT(
    vocab_size=config['vocab_size'],
    block_size=config['block_size'],
    n_layer=config['n_layer'],
    n_head=config['n_head'],
    n_embd=config['n_embd'],
    dropout=config['dropout'],
)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

# Load tokenizer
with open('tokenizer.json') as f:
    tokenizer = json.load(f)

# Encode prompt
prompt = "Once upon a time"
indices = [tokenizer['stoi'].get(c, 0) for c in prompt]
input_ids = torch.tensor([indices], dtype=torch.long)

# Generate
output_ids = model.generate(input_ids, max_new_tokens=200, temperature=0.9, top_k=40)
text = ''.join(tokenizer['itos'][str(i)] for i in output_ids[0].tolist())
print(text)

Training

This model was trained using the project's training pipeline:

python run_model.py --train_model --arch micro_glm --dataset tinystories --max_steps 500

Files in this repository