TinyStories-45M

A 45-million parameter language model trained entirely on the TinyStories dataset for creative story generation. This model follows the LLaMA architecture with grouped query attention (GQA) and is optimized for short-form narrative text.

Model Details

Training

Pretraining

• Dataset: roneneldan/TinyStories
• Epochs: 3
• Effective Batch Size: 128
• Learning Rate: 5e-4 with cosine decay
• Warmup: 1%
• Weight Decay: 0.1
• Precision: FP16
• Optimizer: AdamW

Supervised Fine-Tuning (SFT)

• Dataset: roneneldan/TinyStoriesInstruct
• Epochs: 1
• Learning Rate: 1e-4
• Loss Masking: Assistant-only (only compute loss on story completion)

Tokenizer

• Type: SentencePiece Unigram
• Vocab Size: 16,384
• Special Tokens: <pad>, <eos>, <bos>, <unk>, <|im_end|>

Evaluation

50-Prompt Inference

See evaluation/50_prompts.json for generated story samples.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("razor5050/TinyStories-45M")
tokenizer = AutoTokenizer.from_pretrained("razor5050/TinyStories-45M")

prompt = "Features: a brave cat\nWords: moon, adventure\nSummary: A cat goes on a moon adventure\nStory:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Hardware

• Training GPU: NVIDIA RTX 3060 12GB
• Training Time: ~8-10 hours (pretrain + SFT)

Citation

@dataset{roneneldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Ronen Eldan and Yuanzhi Li},
  year={2023}
}

---

Generated: 2026-05-20 18:37:02