TinyStories 35M (base + chat)

A small GPT (decoder-only transformer) trained from scratch on TinyStories on a single Apple M5 laptop, plus a chat fine-tune ("Tiny", a story bot for kids).

Model

Training

Base (code/train.py, log: logs/train.log)

• Data: 2,119,719 TinyStories training stories (~472M tokens), each followed by <|endoftext|>
• 10,682 steps × 65,536 tokens/step = 700M tokens (~1.5 epochs)
• AdamW (β 0.9/0.95, wd 0.1), lr 1e-3 → 1e-4 cosine, 300 warmup steps, grad clip 1.0, bf16 autocast
• Apple M5 (MPS), ~10.5k tokens/s, ~19 hours
• Final loss: train 1.302, validation 1.340

Chat (code/finetune_chat.py, data builder code/chat_data.py, log: logs/finetune_chat.log)

• Starts from the base model; 3,000 steps, lr 2e-4 → 2e-5
• Synthetic conversations: small talk, story requests (answered with real TinyStories stories), passage QA, simple kid "why" questions, out-of-scope refusals
• Loss only on Assistant turns; every few steps trains on plain TinyStories text to keep storytelling
• Final: chat validation loss 1.048, story loss 1.664 (base: 1.340). The chat model is noticeably weaker as a plain story continuer, so use base for raw story generation.

📄 Technical report

The full write-up of this experiment series (including the revision below) is published on Zenodo: Initialization Scale, Not Value Source — DOI 10.5281/zenodo.23489983 (report.pdf also in this repo).

⚠️ 2026-10-20 revision — multi-seed replication overturned the phase-1 "gain"

Earlier versions of this card reported the single-token pre-pass (v4/v7) as a real −0.013 val-loss improvement, "confirmed" by a 2-seed control, and v7 as significantly preferred in generation. Proper replication overturned this:

• Each run's reported "final val" came from 50 randomly sampled eval batches — a protocol that turns out to carry ±≈0.01 measurement noise (e.g. the seed-777 baseline scored 1.3478 on its own eval but is the best of six models, 1.3260, on a fixed eval set).
• Re-scoring all checkpoints on the identical 400 validation batches (3.28M tokens) and training 3 baseline seeds:

Baseline seed-SD ≈ 0.003; the pre-pass is consistently ~+0.0035 worse than its same-seed baseline. The pre-pass gives no gain. (Missing cells: runs lost to a cloud-account ban before download.)

• A 6-way 300-prompt cross-judge among the seed models came out 48/42/53/52/52/53 — all within chance; v7's earlier generation "wins" did not recur against seed-matched baselines.
• What stands: the init-scale U-curve (effects 10–400× the noise), the harm of vocabulary streams, the pyramid results, and the 125M finding below. Details and the full post-mortem are in the Zenodo report.

New: 125M-scale init comparison (12 layers / 12 heads / 768 dim, same budget)

Two runs differing only in init, scored on the same fixed 400 batches: default normal(0, 0.02) → 1.2325; pi ×0.003 (rms ≈ 0.016) → 1.2029. The smaller init wins by 0.030 (~10× the 35M seed-SD) — consistent with the optimal init scale shrinking as width grows. Preliminary (single runs, 125M baseline re-run pending).

unique-v7 — pi-digit initialization + single-token pre-pass

unique-v7 remains the series' most interesting artifact: a model in which none of the weights start from random numbers — every initial weight comes from the decimal digits of pi — that trains exactly as well as its gaussian-initialized siblings.

What exactly v7 is

Why it is meaningful

1. Initialization needs scale, not randomness. A model initialized from pi — a fixed, public, maximally "non-random" sequence — trains exactly as well as one initialized from gaussian noise. Together with its siblings (v6: same digits ×0.07 → destroyed, loss 2.46; v9: ×0.001 → 1.3517; v11: token-id values, rms ≈0.0004 → 1.4772) it maps a clean U-curve with the optimum at weight rms ≈ 0.02 (at width 384): the source of the initial values never mattered, only their magnitude.
2. Fully reproducible init. Since pi's digits are public and the sign RNG is seeded (1337), v7's starting weights are bit-reproducible by anyone — a property gaussian inits don't naturally give you.
3. Its earlier val-loss and generation "wins" are retracted per the revision above: on fixed-batch eval v7 is indistinguishable from (same-seed) baseline, and its +3 SD generation win did not replicate against seed-matched baselines.

Load it from unique-v7/ (same class/config as base).

Init / phase-1 experiments

Thirteen controlled runs on Kaggle T4x2 asking: does a different weight initialization, a phase-1 pre-pass over the corpus's unique tokens, or a pyramid architecture beat plain training? One variable changes per run; the budget is identical everywhere (10,682 iters × 65,536 tokens/step = 700M tokens). unique-v4 and unique-v7 tie as winners (see the headline section above); all variants named unique-* are downloadable from this repo.

Full scoreboard (best → worst)

Note (2026-10-20): the "final val loss" column uses each run's own 50-random-batch eval, which carries ±≈0.01 measurement noise — differences under ~0.02 in this table are not meaningful (see the revision section above for the fixed-batch numbers). The large effects (streams, v6/v8/v9/v11) stand.

("1 sweep" = the single-token pre-pass: each of the 25,944 unique token ids as its own length-1 context, 8 epochs ≈ 56 steps, before training.)

Finding 1 — initialization needs scale, not randomness

The same training recipe with the init's magnitude swept across four orders of structure sources gives a clean U-curve with its optimum at weight rms ≈ 0.02. The source of the values (pi digits, the tokens' own ids, gaussian noise) never mattered — only their size:

Finding 2 — the single-token pre-pass gives no gain (REVISED 2026-10-20)

The original single-seed ordering (v4 1.3267 < v12 1.3330 < base 1.3396) and its "seed-noise control" were artifacts of the noisy per-run eval protocol. On the fixed 400-batch eval across 3 seeds (table in the revision section above), the pre-pass is consistently ~+0.0035 worse than its same-seed baseline, and the apparent v4/v12/base ordering dissolves into the noise band. What survives: the pre-pass only doesn't hurt in its length-1-context form, while unique-word streams actively hurt (+0.025 to +0.084 — effects far above the noise).

Finding 3 — pyramid blocks are (at best) free, never better

Shrinking each block's attention/MLP dims with depth, at a total parameter count matched to the uniform model exactly:

Hardware-friendly dims completely fix the speed penalty, and the gentler ratio shrinks the quality gap to +0.006 — nearly free, but the "deep layers need less capacity" idea never actually beat the uniform model.

Generation: 300-prompt cross-judged comparisons

Protocol: every model continues the same 300 validation-story openings (fixed seed per prompt); each story is scored by every other model's mean log-prob per token; highest average wins the prompt. Four fields were run:

v7 nominally led the first three fields (once at +3 SD), but in the decisive fourth field — against seed-matched baselines instead of weakened variants — everything is a dead tie, so we treat the earlier +3 SD as a field/protocol artifact, not a model property. Robust separation appears only at the bottom: v11 (−4 SD) and v6 (0/300 in an earlier 3-way). Val-loss differences of ±0.02 are invisible to the judges. Caveat: wins are relative to the field and judge panel, not an absolute quality measure.

HellaSwag: a negative control that stayed negative

All 13 runs — and roneneldan's reference models — sit in a 26.2–26.8% band, within noise of the 25% chance floor and of each other (SE ≈ ±0.44pp; per-2000-question blocks swing 0.23–0.32 for every model in lockstep):

No init, pre-pass, or architecture changed it — expected, since HellaSwag tests world knowledge that TinyStories simply doesn't contain. Included as a sanity check, not a scoreboard.

Results

All outputs are in reports/, produced by run_reports.sh. Generations are sampled, so a re-run gives different text.

HellaSwag is close to chance (25%). That is expected: it tests general world knowledge, while this model has only read children's stories. It is included as a sanity check, not a strength.

Comparison with the original TinyStories models

Benchmarked against roneneldan's original TinyStories models (the paper's reference checkpoints), evaluated locally with identical scoring code.

HellaSwag (full validation set, 10,042 items; code/hellaswag_hf.py for the HF models):

The original models score best here: TinyStories-2Layers-33M is the strongest on both metrics, and TinyStories-8M also edges this model on raw accuracy. That said, all TinyStories-only models sit within ~2 points of chance regardless of size — the corpus simply contains no adult commonsense knowledge, so none of these differences mean much.

300-prompt cross-judged story generation (code/compare_300_hf4way.py): each model continues the same 300 validation-story openings (fixed seed per prompt); every story is scored by the other three models' mean log-prob per token; highest average judge score wins the prompt.

This model is statistically tied with the paper's 8M reference model and clearly ahead of the 2-layer 33M one — at this scale, depth matters more than width.

Example: chat

You : hi
Tiny: Hey! I am Tiny. I can tell stories and answer questions about them.
You : Why do we brush our teeth?
Tiny: We brush our teeth to keep them clean and strong, so they do not hurt.
You : What is the capital of France?
Tiny: I'm sorry, I don't know about that. I know about kids, animals and stories. Shall I tell you a story?
You : I am sad today
Tiny: Oh no, I am sorry you feel sad. Do you want a happy story to cheer you up?

Example: base story (prompt "Once upon a time")

Once upon a time, there was a little girl named Lily. She loved to play in her room with her toys.
One day, she saw her mom cleaning and said, "Mommy, can I help you?" Her mom said, "Sure, Lily.
You can help me wipe the floor."

Example: grammar check (base model scores candidate edits)

IN : the girl want to go hoospital
OUT: the girl wanted to go to the hospital
IN : Tom wanted play with his freind.
OUT: Tom wanted to play with his friends.

All reports

Usage

pip install torch tiktoken safetensors huggingface_hub
# get the code folder, then from inside it:
python from_hub.py --repo sraivante/tinystories-35m           # writes out/final.pt (base) and out/chat.pt (chat)
python sample.py --ckpt out/final.pt --prompt "Once upon a time"
python chat.py                              # interactive chat with Tiny
python grammar_check.py "He have a big red ball."
./start.sh                                  # web UI at http://127.0.0.1:8765

Or load directly in Python (with code/model.py on the path):

import torch, tiktoken
from from_hub import load
model, cfg = load("sraivante/tinystories-35m", "base")        # or "chat"
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode_ordinary("Once upon a time")])
out = model.generate(ids, 200, temperature=0.8, top_k=200, stop_token=enc.eot_token)
print(enc.decode(out[0].tolist()))

Chat prompt format: User: <message>\nAssistant:; stop when the model writes \nUser:.

Limitations

• Knows only the simple world of children's stories; no facts, maths or general knowledge.
• Stories can lose track of characters or contradict themselves, especially past ~300 tokens.
• Passage QA misses questions that need reasoning across sentences (e.g. "What broke the sandcastle?").
• Not safety-tuned beyond the chat model's out-of-scope refusals; not intended for production use.

Files

config.json  configs of both variants (read by from_hub.py)
base/  model.safetensors, config.json
chat/  model.safetensors, config.json
code/  model, training, fine-tuning, evaluation, inference, web UI, from_hub.py
logs/  train.log, finetune_chat.log
reports/  evaluation outputs (see above)
run_reports.sh  regenerates reports/