tosodoulikoGPT (10M)

«τοσοδούλικο»: teeny-tiny. A 10-million-parameter Greek language model trained from scratch: its own tokenizer, its own weights, Greek text only. It is the smallest member of the tosodouli family, built for education and research: a model small enough to train in minutes and read in an afternoon, and a baseline for Greek models.

It is a base model: it continues text. It is not instruction-tuned, not a chatbot, and knows little about the world.

Samples

Prompt «Μια φορά κι έναν καιρό» (in bold), continued by the model with temperature 0.8, top-k 50:

Μια φορά κι έναν καιρό, οι περισσότεροι από εμάς είχαμε την τύχη να έχουμε την τύχη να ξαναπροσπαθήσουμε να κατακτήσουμε τα όνειρά μας και να πάμε σε ένα άλλο μονοπάτι.

Μια φορά κι έναν καιρό, το να είσαι κάτι καινούργιο, ένα καινούργιο κομμάτι, ένα καινούργιο, αλλά ταυτόχρονα και πάλι μαγικό. Έχοντας στο μυαλό σου ένα καινούριο κομμάτι της περιπέτειας, σίγουρα δε θα το έχεις καν χαμένο, οπότε θα καταλάβεις τι σημαίνει να ξεκινάς ένα ταξίδι από το παρελθόν.

Μια φορά κι έναν καιρό, η Εστία Πάτμου ήταν κλειστή μόνο για έξι μήνες, ενώ η ΠΟΕΣΥ έχει κάνει αρκετές επιλογές για τη μείωση του αριθμού των επισκεπτών και τη μείωση της επιστροφής που ενδεχομένως θα ισχύσει μέχρι το τέλος της εβδομάδας.

Use

The weights load as a standard GPT2LMHeadModel. The tokenizer is our own (character-level Greek BPE) and ships as tokenizer.py (standard library only) with tokenizer.json.

import importlib.util, torch
from huggingface_hub import snapshot_download
from transformers import GPT2LMHeadModel

path = snapshot_download("odychlapanis/tosodouliko-gpt")
spec = importlib.util.spec_from_file_location("tokenizer", f"{path}/tokenizer.py")
tokenizer = importlib.util.module_from_spec(spec); spec.loader.exec_module(tokenizer)
bpe = tokenizer.GreekBPE.load(f"{path}/tokenizer.json")
model = GPT2LMHeadModel.from_pretrained(path).eval()

ids = torch.tensor([[bpe.document_id] + bpe.encode("Μια φορά κι έναν καιρό")])   # start with the document token
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50, pad_token_id=0)
print(bpe.decode(out[0, 1:].tolist()))

Notes: begin every input with bpe.document_id (it means "a new document starts"), as in training; a sentence break is its own token (bpe.newline_id); the context is 1,024 tokens.

Model

• Parameters: 10.3M (2.6M of them the embedding table)
• Architecture: GPT-2 / nanoGPT shape: 6 layers, width 320, 5 heads, GELU MLP (4×), pre-norm LayerNorm, learned positions, tied input/output embeddings, no biases in linear layers
• Context: 1,024 tokens
• Tokenizer: 8,000-token character-level BPE trained on our Greek text; word-start marker «▁»; digits split one by one
• Alphabet: closed. The 24 Greek letters in both cases, accented vowels and diaereses, digits and « .,;!:'-«» ». Anything else (Latin letters, symbols, emoji) has no token and is dropped by the tokenizer. The model therefore cannot produce a non-Greek character

Training

• Data: FineWeb-2 Greek (ell_Grek, ODC-BY), three files, kept at sentence level and filtered to sentences written entirely in the alphabet above: 3.6B tokens.
• Run: 105,000 steps × 128 sequences × 1,024 tokens ≈ 13.8B tokens (about 3.8 passes over the data), AdamW (β 0.9/0.95, weight decay 0.1), peak learning rate 2e-3 with warmup and cosine decay, no dropout. The weights are from step 103,000, the lowest validation loss.

Evaluation

All scores are computed from the model's probabilities (no generation), zero-shot.

Greek school benchmark (unreleased). Every item is taken from a Primary (Δημοτικό) or Junior high (Γυμνάσιο) school book:

• Grammar: a minimal pair between the book's sentence and a version with one grammatical error. Chance 50%.
• Knowledge: a true statement from a History, Geography, Science or Environment book, among versions that differ in one fact. Chance ≈ 28%.
• Test split: Primary 361 grammar + 518 knowledge items; Junior 169 + 498.
• School score: each task rescaled so 0 = chance and 100 = perfect, then averaged (as nanochat's CORE).

All benchmarks are the Greek versions.

• Chars / token: characters per token on the 798 FLORES Greek sentences used for FLORES bpb, each tokenized on its own (higher is better: the same Greek costs fewer tokens).
• Public benchmarks: MultiBLiMP Greek (1,096 grammatical / ungrammatical pairs); ARC-Easy Greek (ILSP translation, 2,376 questions) and HellaSwag Greek (ILSP translation, 10,024 contexts), both in cloze form, accuracy with each answer's loss divided by its length in characters (acc_norm); FLORES bpb: bits per byte on the 798 FLORES-200 Greek devtest sentences written fully in the closed alphabet (lower is better; comparable across tokenizers).
• School: the school score, centered (0 = chance, 100 = perfect); G / K are grammar and knowledge accuracies.

At 10M parameters the model has learned Greek grammar well (MultiBLiMP, school grammar) and almost no knowledge: ARC and HellaSwag are near chance.

Limitations

• A small base model: it writes fluent Greek with drifting meaning and will state false things confidently. Do not use it for facts or advice.
• Trained on web text, with the biases of web text; no safety tuning.
• Greek only, closed alphabet: Latin letters, most symbols and emoji are silently dropped by the tokenizer.
• 1,024-token context.

Family

• tosodoulikoGPT (10M): this model.
• tosodouliGPT (110M): odychlapanis/tosodouli-gpt.

Code (training, tokenizer, evaluation) will be published on GitHub.

Citation

@misc{chlapanis2026tosodoulikogpt,
  author       = {Chlapanis, Odysseas S.},
  title        = {tosodoulikoGPT: a 10M-parameter Greek language model trained from scratch},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/odychlapanis/tosodouliko-gpt}}
}

Acknowledgements

• Training data: FineWeb-2 (Hugging Face, ODC-BY).
• Architecture template: nanoGPT (Andrej Karpathy).
• Teaching approach: Build a Large Language Model (From Scratch) (Sebastian Raschka).
• Evaluation: MultiBLiMP (Jumelet et al.); Greek ARC and HellaSwag translations and FLORES-200 Greek (ILSP / Athena Research Center); FLORES-200 (NLLB Team, Meta AI).

Licence

MIT. The training data (FineWeb-2) is ODC-BY and is not redistributed.

Author: Odysseas S. Chlapanis.