Tiny GPT (5.8M parameters)

A small GPT-2-style decoder-only transformer, written from scratch in PyTorch (no transformers classes), trained on a laptop CPU on a subset of TinyStories. It continues short children's stories. Code, training logs and the full write-up are in the GitHub repository.

Model details


Architecture	pre-norm GPT, 6 layers, d_model 256, 8 heads, context 256, learned positions, tied embeddings
Parameters	5,853,184
Tokenizer	byte-level BPE, vocabulary 4096, trained on the training split only (`tokenizer.json`)
Files	`model.safetensors`, `model_config.json` (our `ModelConfig`), `tokenizer.json`, `model_info.json`

Training

Data: first 300,000 stories (60.8M tokens) of TinyStories V2 (GPT-4 generated); validation on 5,000 held-out stories.
Run: 9,500 steps x 8,192 tokens = 77.8M tokens (about 1.28 passes over the data), AdamW, peak lr 1e-3 with warmup and cosine decay, 12 CPU threads, 7.9 hours of running time on one laptop (Intel Core Ultra 9 185H).

Evaluation

Model	Validation loss	Perplexity
Uniform guess	8.32	4096
Unigram counts	5.88	357
Bigram counts	3.65	38.4
This model	1.688	5.41

Perplexity is per BPE token with this 4096-token tokenizer, on TinyStories text, so it is not comparable to numbers from other tokenizers or datasets.

Intended use

Learning, demos and experiments with small language models; generating simple children's-story-style text.

Limitations and risks

Narrow domain. It only knows the simple vocabulary and plots of TinyStories. It cannot follow instructions, answer questions or hold a conversation.
Quality. Stories are fluent at sentence level but repeat phrases, lose objects and plot threads, and sometimes contradict themselves. Samples in the repository's docs/RESULTS.md include failures, not just good examples.
Synthetic data. The training stories were written by a large language model, so the model inherits that model's stylistic and social biases (names, gender roles, settings). It has not been evaluated for bias or safety.
Not for factual content, advice, or anything where wrong or biased text could cause harm.

Data and licence

Code and weights in this repository are released under the MIT licence. The training data, TinyStories by Ronen Eldan and Yuanzhi Li, is licensed CDLA-Sharing-1.0 on the dataset card; please credit it and check the dataset card for current terms.

Usage

git clone <github repo> && cd tiny-gpt && make setup
TINYGPT_HF_REPO=<this repo id> TINYGPT_MODEL_DIR=./model uv run uvicorn tinygpt.serving.app:create_app --factory

Loading…

Model details


Architecture	pre-norm GPT, 6 layers, d_model 256, 8 heads, context 256, learned positions, tied embeddings
Parameters	5,853,184
Tokenizer	byte-level BPE, vocabulary 4096, trained on the training split only (`tokenizer.json`)
Files	`model.safetensors`, `model_config.json` (our `ModelConfig`), `tokenizer.json`, `model_info.json`

Training

Data: first 300,000 stories (60.8M tokens) of TinyStories V2 (GPT-4 generated); validation on 5,000 held-out stories.

Run: 9,500 steps x 8,192 tokens = 77.8M tokens (about 1.28 passes over the data), AdamW, peak lr 1e-3 with warmup and cosine decay, 12 CPU threads, 7.9 hours of running time on one laptop (Intel Core Ultra 9 185H).

Model

Validation loss

Perplexity

Uniform guess

8.32

4096

Unigram counts

5.88

357

Bigram counts

3.65

38.4

This model

1.688

5.41

Limitations and risks

Narrow domain. It only knows the simple vocabulary and plots of TinyStories. It cannot follow instructions, answer questions or hold a conversation.

Quality. Stories are fluent at sentence level but repeat phrases, lose objects and plot threads, and sometimes contradict themselves. Samples in the repository's docs/RESULTS.md include failures, not just good examples.

Synthetic data. The training stories were written by a large language model, so the model inherits that model's stylistic and social biases (names, gender roles, settings). It has not been evaluated for bias or safety.

Not for factual content, advice, or anything where wrong or biased text could cause harm.

Ahmadbader17/tiny-gpt

Tiny GPT (5.8M parameters)

Model details

Training

Evaluation

Intended use

Limitations and risks

Data and licence

Usage

Tiny GPT (5.8M parameters)

Model details

Training

Evaluation

Intended use

Limitations and risks

Data and licence

Usage