Char-GPT 1.2M

A tiny character-level causal transformer trained from scratch on TinyStories. A small, honest reference build — the point is a model whose card matches its artifact exactly, not a competitive checkpoint.

Parameters (exact)

1,216,000 parameters, untied head.

The head is not weight-tied: the checkpoint stores two distinct 65×128 tensors (transformer.wte.weight and lm_head.weight), and model.py never assigns one to the other. config.json therefore says tie_word_embeddings: false. (If the head were tied the count would be 1,207,680.)

Architecture

nanoGPT-style GPT-2, all bias-free except LayerNorm:

• n_layer=6, n_head=4, n_embd=128, FFN = 4× = 512
• vocab_size=65 (printable ASCII + newline), block_size=128
• RoPE: none (learned positional embedding wpe)

Training

• Data: roneneldan/TinyStories (train split), first ~1.0M characters, 90/5/5 train/val/test split by character.
• Steps: 1,500, batch 32 × seq 128, AdamW (lr 6e-4, cosine, warmup), grad-clip 1.0, float32, CPU (16 threads). ~3.5 min.
• Seed: 42.

Quality — what it is and is not

Held-out perplexities (measured on the full held-out test split, 2026-09-20):

It captures TinyStories' surface style (short declarative sentences, simple vocabulary, character names) but it is a 1.2M-parameter model on ~1M characters — it does not grasp meaning, it repeats and drifts, and it will produce the kind of plausible-looking-but-nonsense text in sample.txt. Treat it as a working toy / reference architecture, not a useful language model.

The original training log reported val 1.9046 / test 1.9473 from a 60-batch random evaluation; the full-split number above is the honest one.

Files

• model.safetensors — 4,869,112 B (53 tensors, F32)
• model.py — CharGPT + from_config
• config.json, tokenizer_config.json (char vocab)
• sample.txt — 240-char greedy-ish sample
• LICENSE — Apache-2.0

Reproduce

import torch, json
from model import from_config
cfg = json.load(open("config.json"))
m = from_config(cfg)
print(sum(p.numel() for p in m.parameters()))  # 1216000