加载中…
加载中…
TinyStories-17M is a 17.2 million parameter decoder-only Transformer trained entirely from scratch for children's story generation.
The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts.
This project is inspired by the TinyStories paper:
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Ronen Eldan, Yuanzhi Li (2023)
Paper: https://arxiv.org/abs/2305.07759
Dataset: https://huggingface.co/datasets/roneneldan/TinyStories
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer |
| Parameters | 17,234,304 |
| Layers | 8 |
| Hidden Size | 384 |
| Attention Heads | 6 |
| Feed Forward Size | 1024 |
| Context Length | 512 |
| Vocabulary Size | 8000 |
| Positional Encoding | Rotary Embeddings (RoPE) |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Weight Tying | Yes |
The model was trained from scratch using next-token prediction on a synthetic TinyStories-style dataset.
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Precision | BF16 Mixed Precision |
| Learning Rate | 3e-4 |
| Weight Decay | 0.1 |
| Epochs | 8 |
| Batch Size | 70 |
| Context Length | 512 |
| Vocabulary | 8000 |
Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule.
The model was trained on approximately 2.1 million synthetic English children's stories inspired by the TinyStories dataset.
The dataset focuses on:
TinyStories-17M is intended for:
The model was evaluated using a combination of:
Evaluation focused on:
| Category | Score |
|---|---|
| Grammar | 9.7 / 10 |
| Story Coherence | 8.7 / 10 |
| Character Consistency | 8.6 / 10 |
| Creativity | 8.1 / 10 |
| Repetition Resistance | 8.0 / 10 |
| Overall Story Quality | 8.62 / 10 |
Note: These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results.
The following examples are actual generations produced by TinyStories-17M.
Once upon a time
there was a little girl who was very curious. One day she decided to
explore the woods and take a look around. As she walked, the little
girl heard some strange noises coming from behind some trees.She
stopped and looked around but couldn't see anything. Suddenly, a
voice said, "Help me!" The little girl was scared but she asked,
"Who are you?" The voice replied, "I am a giant. I need to tell you
that you must leave these woods and not come back." The little girl
knew she had to do something...
Ben discovered an old compass hidden inside his grandfather's attic.
He decided to explore the attic, but it was very dark and scary. He was
scared, but he was determined to find out what was behind the boxes. He
opened one of the boxes and found a heavy chest. He tried to lift it,
but it was too heavy. He wondered what could be inside. Suddenly, he
heard a mysterious voice introducing itself as the spirit of the attic,
encouraging Ben to continue exploring and discover new adventures...
A lonely dragon met a tiny mouse for the first time.
The dragon wanted to play with the mouse, but the mouse was scared and ran
away. Hoping to become friends, the dragon searched for the tiny mouse
throughout the forest. Soon a misunderstanding turned into an exciting
chase, and both of them learned that first impressions are not always true...
Note: The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(
"kaushik-harsh-99/TinyStories-17M",
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
"kaushik-harsh-99/TinyStories-17M",
trust_remote_code=True,
)
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=200,
temperature=0.8,
top_p=0.95,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
model.safetensors
config.json
generation_config.json
configuration_tiny.py
modeling_tiny.py
__init__.py
tokenizer.json
tokenizer.model
tokenizer_config.json
special_tokens_map.json
README.md
LICENSE
As a 17M parameter language model, TinyStories-17M has several limitations:
The next version of this project focuses on improving both the dataset and training methodology.
Planned improvements include:
This project was inspired by:
Special thanks to the open-source AI community for making efficient language model research accessible.