tiny-llm TinyStories 30M

A 29.9M-parameter Llama-style language model, trained from scratch on a laptop GPU as part of tiny-llm: an LLM training and inference project written from scratch.

It is a base model for story continuation. Give it the opening of a short children's story and it writes the rest, in the simple English of the TinyStories dataset. It does not follow instructions or chat, and it knows nothing outside that kind of story.

Use it

With Hugging Face Transformers (the weights are a standard LlamaForCausalLM):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CoolDip00/tiny-llm-tinystories-30m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Once upon a time, a little dog", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=120, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0]))

Or with the project's own inference server (OpenAI-compatible /v1/completions), on CPU with Docker:

docker build -t tinyllm https://github.com/cooldip00/tiny-llm.git
docker run --rm -p 7860:7860 tinyllm
curl localhost:7860/v1/completions -H 'content-type: application/json' \
     -d '{"prompt": "Once upon a time", "max_tokens": 80}'

Sample, prompted with "Once upon a time": "…a little girl named Lily. She was very curious… she found a shiny rock that looked like it was magic… she saw a butterfly that was also very shiny…"

Model


Architecture	Llama-style decoder: RMSNorm (pre-norm), RoPE (θ = 10,000), SwiGLU MLP, multi-head attention, tied input / output embeddings
Size	29,893,120 parameters: 8 layers, d_model 512, 8 heads (head dim 64), MLP hidden 1,408
Context	512 tokens
Tokenizer	Byte-level BPE, 8,192 tokens, trained on TinyStories. No BOS token; `<
Weights	`model.safetensors`, fp32, as trained

Training


Data	TinyStories train split, cleaned (broken encodings fixed, non-ASCII stories dropped); 996M tokens, ~2.1 epochs
Steps	7,600 × 131,072 tokens (batch 32 × 512, 8 gradient-accumulation steps)
Optimizer	AdamW (β 0.9 / 0.95, weight decay 0.1 on matrices and embeddings), peak LR 1e-3 after 200 warmup steps, cosine decay to 1e-4, gradient clipping 1.0
Precision	bf16 autocast, fp32 master weights, `torch.compile`
Hardware	One laptop RTX 3070 Ti (8 GB): 2 h 27 min, ~115K tokens/s, 56% MFU

Evaluation


Validation loss	1.2825 nats / token on the TinyStories validation split (perplexity 3.6); train loss 1.280, so no overfitting

This is the only metric: the model is too small for general benchmarks to mean anything.

Export check (scripts/export_hf.py in the project): the files in this repo were loaded back and compared on CPU in fp32.

HF Transformers' logits match the project's own model to within 1.4e-5 over 2,741 positions, both as one full forward and as prefill plus cached decode.
Greedy generation matches token for token on 8 prompts, each run to EOS or 200 tokens.
AutoTokenizer gives the project tokenizer's ids on every validation story.

Limitations

Story continuation only, in TinyStories' vocabulary and style. Other prompts (questions, code, other languages) give story-like or meaningless text.
Small and trained on synthetic stories: its text can be repetitive or contradict itself, and none of it is factual.
Inherits whatever biases the TinyStories stories (generated by GPT-3.5 and GPT-4) contain.
Not for any real-world or safety-relevant use; it is a learning and demo model.

License

The weights and code are MIT. The training data, TinyStories, is published under CDLA-Sharing-1.0.

CoolDip00/tiny-llm-tinystories-30m