TinyTurk Wiki - A_tiny (A (tiny))

Tiny, efficient - 1.38M params

This model is part of the TinyTurk research project on architectural design for tiny-scale Turkish language modeling. See companion models:

• TinyTurk-Wiki-A-Tiny
• TinyTurk-Wiki-B-Sweet
• TinyTurk-Wiki-C-Max

Research Repository

This model is part of the TinyTurk research project. Full source code, training scripts, ablation studies, and evaluation tools are available at:

https://github.com/stunmuffin/TinyTurk

Model Summary

Training

Trained from scratch on a 49,000-article subset of Turkish Wikipedia (barandinho/wikipedia_tr), tokenized with a custom 1024-vocab BPE tokenizer (UNK rate 0.078%).

Evaluation

In-Domain (Wikipedia validation, 945 samples)

Out-of-Domain (Turkish structural diagnostics)

See the GitHub repository for full results. Key observation: structural diagnostics peak at the 2.76M (B) configuration and degrade at larger scales, revealing a task-dependent optimum.

Architecture Notes

This model family tests three hypotheses departing from standard transformer design:

1. Shallowness dominates depth - 2 layers outperform 3, 4, 6 at matched budget.
2. Compressed FFN ratios - FFN = 1.3-2.2x d_model, not 4x.
3. Width scales monotonically - with diminishing returns, following PPL proportional to N^-0.23.

Usage

Install:

pip install torch sentencepiece huggingface_hub

Python:

import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from model import TinyTurkGPTV27, TinyTurkV27Config

REPO = "stunmuffin/TinyTurk-Wiki-A-Tiny"

ckpt_path = hf_hub_download(REPO, "pytorch_model.bin")
tok_path = hf_hub_download(REPO, "tokenizer_wiki_bpe_1024.model")

ckpt = torch.load(ckpt_path, map_location="cpu")
config = TinyTurkV27Config(**ckpt["config"])
model = TinyTurkGPTV27(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

sp = spm.SentencePieceProcessor(model_file=tok_path)

prompt = "Türkiye, "
ids = [sp.bos_id()] + sp.encode(prompt, out_type=int)
input_ids = torch.tensor([ids])

with torch.no_grad():
    out = model.generate(input_ids, max_new_tokens=200,
                         temperature=0.7, top_k=40)

print(sp.decode(out[0].tolist()))

See example.py for a runnable script.

Limitations

• Wikipedia domain only. Performance on narrative, dialogue, or code is degraded.
• Small vocabulary (1024 BPE) - rare words may be split.
• Context 384 tokens. Not suitable for long documents.
• Raw pretrained LM. No instruction tuning.
• No safety alignment.

Citation

@misc{tinyturk_a_tiny_2026,
  title  = {TinyTurk Wiki - A_tiny: Turkish 1.38M parameter LM},
  author = {TinyTurk Project},
  year   = {2026},
  url    = {https://huggingface.co/stunmuffin/TinyTurk-Wiki-A-Tiny}
}

TinyTurk is an experimental research artifact. Not a product.