TinyTurk v2.7d MID - English

20M-parameter English GPT. Reaches 97% of GPT-2 small's BLiMP score with 6.2x fewer parameters.

Model Description

TinyTurk MID is the middle-scale model in the TinyTurk series. It is trained from scratch on a 22 GB mixed English corpus and evaluated against GPT-2 small.

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: English
• License: Apache 2.0

Architecture

• Tokenizer: 16,384 BPE (SentencePiece, English)
• Context length: 256
• Embedding: 384
• Layers: 6
• Heads: 6 (head_dim=64)
• FFN: 384 -> 1536 -> 384
• Micro-FFN: 384 -> 768 -> 384 (gated, gate=0.75)
• Position encoding: RoPE
• Normalization: RMSNorm (pre-norm)
• Weight tying: yes
• Parameters: 20,482,176

Training

Data

A fresh 16K BPE tokenizer was trained on this corpus, then the text was chunked into 256-token pieces.

Optimization

• Training samples used: 1,000,000 chunks (256M tokens, 4.6% of available data)
• Batch size: 32
• LR: 3e-4 cosine, 2% warmup
• Epochs: 1
• Hardware: GTX 1080 Ti
• Time: ~2.2 hours
• Best val loss: 3.4537
• Val perplexity: 31.62

Evaluation

BLiMP (full test, 67,000 items)

97% of GPT-2 small's BLiMP score with 16% of the parameters.

HellaSwag (200 items)

ARC-Easy (100 items)

Language modeling

• Val loss: 3.4537
• Perplexity: 31.62

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_16k.model')

config = TinyTurkV27Config(vocab_size=16384, context_length=256,
    d_model=384, n_heads=6, n_layers=6, d_ff=1536)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'The capital of France is'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(100):
        logits, _ = model(x[:, -256:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

• Generation quality is poor. Despite strong benchmark scores, the model does not produce coherent text on short prompts. This is because the mixed training corpus does not have a clear register.
• Trained on only 4.6% of available data.
• Context length 256 tokens (vs GPT-2's 1024).
• Vocab 16,384 (vs GPT-2's 50,257).
• Below GPT-2 small on ARC-Easy (knowledge-heavy).

Scientific Context

This model tests whether modern architectural choices (RoPE, RMSNorm, micro-FFN, weight tying) plus Chinchilla-optimal training can close the gap to GPT-2 small at a fraction of the size.

Result: At the 20M parameter scale, modern architectures + efficient training achieve near-GPT-2-small grammar performance with 16% of the parameters. However, generation quality is much worse - benchmarks do not tell the whole story.

Files

Citation

@misc{tinyturk-v27d-mid-en,
  title = {TinyTurk v2.7d MID - English},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-mid-en}
}

Related

• Turkish agent: stunmuffin/tinyturk-v2.7d-agent
• English agent: stunmuffin/tinyturk-v2.7d-agent-en
• English TinyStories: stunmuffin/tinyturk-v2.7d-tinystories-en
• Turkish baseline: stunmuffin/tinyturk-v2.7d-k-hikaye-baseline