TinyTurk v2.7d - TinyStories EN

1.1M-parameter English GPT trained from scratch on 200,000 TinyStories.

Model Description

TinyTurk is a family of extremely small language models (~1M parameters). This is the English variant, trained on Microsoft's TinyStories dataset. It uses the same architecture as the Turkish k_hikaye models, but with an English-specific 512 BPE tokenizer and 26x more training data.

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: English
• License: Apache 2.0
• Variant: Baseline (no distillation)

Architecture

• Tokenizer: 512 BPE (SentencePiece, English)
• Context length: 128
• Embedding: 128
• Layers: 4
• Heads: 4
• FFN: 128 -> 512 -> 128
• Micro-FFN: 128 -> 256 -> 128 (gated, gate=0.75)
• Position encoding: RoPE
• Normalization: RMSNorm (pre-norm)
• Weight tying: yes (embedding <-> LM head)
• Activation: GELU
• Dropout: 0.10

Training

• Data: roneneldan/TinyStories - 200,000 stories (train), 10,000 (val), 10,000 (test)
• Optimizer: AdamW (wd=0.01)
• Learning rate: 3e-4, cosine decay, 5% warmup
• Batch size: 32
• Epochs: 30 (stopped early; best at epoch 29)
• Hardware: NVIDIA GTX 1080 Ti
• Training time: ~1.5 hours
• Best val loss: 1.3349
• Perplexity: 3.80

Evaluation

Language modeling

Test loss is lower than val loss - no overfitting.

BLiMP (English, 67 phenomena, 50 items each = 3,350 items, chance=50%)

• Accuracy: 55.61%
• Gap over chance: +5.61 points

Best phenomena

Worst phenomena (below chance)

Layer-wise Analysis (BLiMP)

We extract hidden states after each Transformer block, apply the final RMSNorm + LM head, and run BLiMP at each level. This reveals where in the network English grammatical capability emerges.

BLiMP accuracy by layer

Visual

  emb ########################## 52.36%
   L1 #########################  51.31%  <- dip
   L2 ########################## 52.51%
   L3 ########################## 53.73%
   L4 ########################### 55.61%

Key findings

1. Embedding already scores 52.36% - well above chance. Many BLiMP phenomena are solvable from surface cues (e.g., determiner-noun agreement).
2. L1 causes a dip (52.36% -> 51.31%). The first Transformer layer degrades the useful embedding signal before other layers recover it.
3. Gradual, monotonic improvement from L1 to L4. Each layer adds ~1-2 percentage points.
4. L4 is still improving. The model has not saturated at 4 layers.

Comparison: TR vs EN layer-wise profile

Same architecture, different languages. Layer-wise diagnostics reveal fundamentally different learning profiles:

Interpretation: Turkish exhibits a threshold phenomenon - layers 1-2 do nothing grammatical, grammar begins at L3. English shows gradual development starting from the embedding layer. This is likely because Turkish's agglutinative morphology requires deeper transformations, while English syntax is more surface-level.

Note: The Turkish and English benchmarks are different (TurBLiMP vs BLiMP, cloze vs minimal-pair). Direct numerical comparison is not valid; the comparison here is about the shape of layer-wise development.

Generation Samples

Sampling: temperature=0.8, top_k=40, seed=42

Prompt: 'Once upon a time'

Once upon a time, in a big, green forest, there lived a jolly monkey. He liked to play with his friends and play with his friends. One day, he was walking in the forest when he saw a big, scary snake. The snake was very mean and kept flying around.

Prompt: 'In a small village'

In a small village, there was a little girl named Lily. She loved to play with her teddy bear. One day, Lily went for a walk in the park with her mom. The sun was shining, and the birds flew high in the sky. Lily saw a lot of dust. She wanted to see the dust too.

Greedy (temperature=0.01, top_k=1)

Prompt: 'Once upon a time'

Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red ball in the grass. She wanted to play with it, but it was too high for her to reach. Lily's mom saw her and said, "Lily, you need to reach the ball. It's too high up in the sky." Lily was

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_512.model')

config = TinyTurkV27Config(vocab_size=512, context_length=128)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'Once upon a time'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(80):
        logits, _ = model(x[:, -128:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

• Extremely small (1.1M parameters) - limited generation quality
• Trained on simple children's stories - narrow domain
• Context window: 128 tokens
• English only
• BLiMP 55.61% - close to chance level, fails on abstract syntax (NPI, island, gaps)
• Occasional semantic drift and repetitions

Scientific Context

This model is part of a broader comparison: same architecture, different language and data scale. It shows that for tiny models, data quantity matters more than architectural tweaks - but only up to a point.

Ongoing/future experiments:

• Turkish TinyStories (same data scale, TR)
• English k_hikaye-sized (7.7K, EN)
• Teacher-Augmented (KD) on EN
• Depth scaling: 4 -> 6 -> 8 layers

Citation

@misc{tinyturk-v27d-tinystories-en,
  title = {TinyTurk v2.7d - TinyStories EN},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-tinystories-en}
}

Related

• Turkish baseline (k_hikaye): stunmuffin/tinyturk-v2.7d-k-hikaye-baseline
• Turkish TAT variant: stunmuffin/tinyturk-v2.7d-k-hikaye-tat
• Dataset: roneneldan/TinyStories