TinyTurk v2.7d — k_hikaye TAT (Teacher-Augmented)

1.1M-parameter Turkish GPT trained with output-level knowledge distillation from a 151M local teacher.

Model Description

TinyTurk is a family of extremely small Turkish language models (~1M parameters). This variant is trained with Teacher-Augmented Training (TAT): the 151M-parameter Turkish model ufakai/ufakzeka-1-base provides soft top-K distributions for the last position of each story.

Research finding: In a 1M-parameter model, output-KD is neutral on val loss, harmful on top-1 diagnostic and TurBLiMP, and slightly beneficial on top-5 diagnostic in structural categories.

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: Turkish
• License: Apache 2.0
• Variant: Teacher-Augmented (TAT)
• Teacher: ufakai/ufakzeka-1-base (151M, Turkish)

Architecture

• Tokenizer: 512 BPE (SentencePiece)
• Context length: 128
• Embedding: 128
• Layers: 4
• Heads: 4
• FFN: 128 → 512 → 128
• Micro-FFN: 128 → 256 → 128 (gated, gate=0.75)
• Position encoding: RoPE
• Normalization: RMSNorm (pre-norm)
• Weight tying: yes (embedding ↔ LM head)
• Activation: GELU
• Dropout: 0.10

Training

• Data: k_hikaye — 9,617 mini-stories (cleaned, deduplicated)
• Split: 7,693 train / 961 val / 963 test (80/10/10, seed=42)
• Optimizer: AdamW (wd=0.01)
• Learning rate: 3e-4, cosine decay, 5% warmup
• Batch size: 32
• Epochs: 100
• Hardware: NVIDIA GTX 1080 Ti
• Training time: 877 seconds (~15 min)
• Best val loss: 1.5566
• Perplexity: 4.74

Distillation Setup

• Teacher: ufakai/ufakzeka-1-base (151M, Turkish)
• Teacher inference: 9,617 stories × top-20 distribution (10 min on CUDA)
• Tokenizer mapping: 40K teacher vocab → 512 TinyTurk vocab
• KD signal: last position only
• Temperature: 1.5
• Top-K: 20
• KD weight schedule:
  • Epoch 0–19: kw = 0.00 (baseline warmup)
  • Epoch 20–59: kw = 0.05
  • Epoch 60–79: kw = 0.02
  • Epoch 80–100: kw = 0.00 (independent finish)

Evaluation

TurBLiMP (16,000 items, chance=50%)

• Accuracy: 65.27%
• Gap over chance: +15.27 points

Comparison with baseline

Language modeling

• Val loss: 1.5566
• Perplexity: 4.74

Grammar diagnostic (456 items, cloze)

By category (Top-5)

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load("tokenizer_v11_bpe_512.model")

config = TinyTurkV27Config(vocab_size=512, context_length=128)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

prompt = "Ali bahçeye"
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(30):
        logits, _ = model(x)
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

• Extremely small (1.1M parameters) — limited generation quality
• Trained on simple children's stories — narrow domain
• Context window: 128 tokens
• Turkish only
• KD hurts top-1 accuracy and TurBLiMP — do not use for single-best-prediction tasks
• KD damages irregular morphology (-6.50 on TurBLiMP) — likely due to broken tokenizer mapping

Scientific Context

This model tests whether output-level KD helps extremely small LMs. Result: mixed-to-negative.

Hypotheses for failure:

1. Broken tokenizer mapping (40K → 512)
2. Insufficient capacity (1M params for two signals)
3. Single-position signal (last token only)
4. Teacher-model mismatch (151M vs 1M)

Citation

@misc{tinyturk-v27d-k-hikaye-tat,
  title = {TinyTurk v2.7d — k_hikaye TAT},
  author = {stunmuffin},
  year = {2026},
  howpublished = {\url{https://huggingface.co/stunmuffin/tinyturk-v2.7d-k-hikaye-tat}}
}

Related

• Baseline variant: stunmuffin/tinyturk-v2.7d-k-hikaye-baseline
• Teacher model: ufakai/ufakzeka-1-base