TinyTurk v2.7d — k_hikaye Baseline

1.1M-parameter Turkish GPT trained from scratch on 7,693 mini-stories.

Model Description

TinyTurk is a family of extremely small Turkish language models (~1M parameters) designed to test how far a tiny transformer can go on Turkish morphology and syntax. This version is trained on 7,693 short stories (k_hikaye dataset).

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: Turkish
• License: Apache 2.0
• Variant: Baseline (no distillation)

Architecture

• Tokenizer: 512 BPE (SentencePiece)
• Context length: 128
• Embedding: 128
• Layers: 4
• Heads: 4
• FFN: 128 -> 512 -> 128
• Micro-FFN: 128 -> 256 -> 128 (gated, gate=0.75)
• Position encoding: RoPE
• Normalization: RMSNorm (pre-norm)
• Weight tying: yes (embedding <-> LM head)
• Activation: GELU
• Dropout: 0.10

Training

• Data: k_hikaye - 9,617 mini-stories (cleaned, deduplicated)
• Split: 7,693 train / 961 val / 963 test (80/10/10, seed=42)
• Optimizer: AdamW (wd=0.01)
• Learning rate: 3e-4, cosine decay, 5% warmup
• Batch size: 32
• Epochs: 100
• Hardware: NVIDIA GTX 1080 Ti
• Training time: 676 seconds (~11 min)
• Best val loss: 1.5562
• Perplexity: 4.74

Evaluation

TurBLiMP (16,000 items, chance=50%)

• Accuracy: 66.86%
• Gap over chance: +16.86 points

By phenomenon

Language modeling

• Val loss: 1.5562
• Perplexity: 4.74

Grammar diagnostic (456 items, cloze)

By category (Top-5)

Layer-wise Analysis

We extract hidden states after each Transformer block, apply the final RMSNorm + LM head, and run the 456-item diagnostic at each level. This reveals where in the network each linguistic capability emerges.

Top-1 accuracy by layer

Top-5 accuracy by layer

Key findings

1. Embedding, L1, L2 produce zero linguistic accuracy (Top-1). The first two layers do not perform grammatical tasks - only surface feature extraction.
2. L3 is the critical layer. All four categories jump from near-zero to meaningful accuracy at L3.
3. L4 continues to develop all categories. The model is still learning at the last layer.
4. Long-distance and word-order remain weak at L4 (6.88% and 2.08%), suggesting these may require more depth.

Interpretation: TinyTurk exhibits a threshold phenomenon, not gradual specialization. Layers 1-2 are essentially feature extractors; grammar begins at L3 and develops at L4. This provides a scientific rationale for future depth scaling experiments (4 -> 6 -> 8 layers).

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_v11_bpe_512.model')

config = TinyTurkV27Config(vocab_size=512, context_length=128)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'Ali bahçeye'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(30):
        logits, _ = model(x)
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Depth Scaling

We trained TinyTurk v2.7d with 4, 6, and 8 layers (same architecture, data, hyperparameters; only depth varies).

Val loss

Val loss follows a U-shaped curve: 6L is the sweet spot, 8L is worse than 4L.

Layer-wise Top-1 diagnostic (final layer)

Key findings

1. No single winner. Each linguistic category has its own optimal depth:
   • local_syntax, long_distance -> 4L (shallow)
   • word_order -> 6L (medium)
   • morphology -> 8L (deep)
2. Depth scaling is not monotonic. Val loss U-curve.
3. Late-layer forgetting. In 8L, local_syntax peaks at L6 (10%) and drops at L8 (8.75%).
4. Morphology is the only category that monotonically scales with depth.

For a general-purpose TinyTurk at this scale, 6L is the best compromise. Full report: DEPTH_SCALING_REPORT.md

Known Limitations

• Extremely small (1.1M parameters) - limited generation quality
• Trained on simple children's stories - narrow domain
• Context window: 128 tokens
• Turkish only
• Weak on ellipsis (13.10%), suspended affixation (32.20%), and subject agreement (35.80%) - far below or near chance level
• Long-distance dependencies and word order are still underdeveloped at L4

Scientific Context

This model is the baseline for the Teacher-Augmented TinyTurk (TAT) experiment, which tests whether output-level KD helps extremely small LMs.

Result of the TAT experiment: mixed-to-negative. KD did not improve the baseline.

Interpretation: For 1M-parameter models, output-level KD is net harmful - likely due to broken tokenizer mapping, insufficient capacity, and single-position teacher signals.

Citation

@misc{tinyturk-v27d-k-hikaye-baseline,
  title = {TinyTurk v2.7d - k_hikaye Baseline},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-k-hikaye-baseline}
}

Related

• TAT variant (Teacher-Augmented): stunmuffin/tinyturk-v2.7d-k-hikaye-tat
• English variant (TinyStories): stunmuffin/tinyturk-v2.7d-tinystories-en
• Teacher model: ufakai/ufakzeka-1-base