Puki Brains (English)

The small language model that writes the English lines of Puki the parrot in Puki, a free Hebrew/English game for toddlers. It runs on the tablet itself, offline.

It does one job: it reads a short prompt about a moment in the game ("the child fed the elephant"), what the child said, or a game's own task (a silly mistake to make on purpose, a drawing to talk about, the day to retell at bedtime, a friend who needs help, the next sentence of a story), and writes one short, kind line for a two-to-four-year-old. It is not a general chat model and knows very little about the world. Its Hebrew sibling is dtsj89/puki-brains-he.

Model

A llama-style decoder of 44M parameters: 14 layers, width 512, 8 attention heads with 2 key/value heads (head size 64), RoPE, tied embeddings, a 512-token context, and an 8,192-token byte-level BPE trained on English.

Training

1. Pretraining from scratch: 1.6B tokens of English (the sources below). Muon (lr 2e-3) with a warmup-stable-decay schedule: a constant phase, then a 1-sqrt cooldown over the last 20%.
2. Fine-tuning: 30,799 of Puki's English lines over a dozen tasks, two epochs. The lines were written by DeepSeek models and kept only if they passed Puki's filters and a second model's judgment (dtsj89/puki-lines).
3. Preference tuning (DPO): 1,485 pairs of its own lines, the better and the worse of four for a prompt as DeepSeek's editor scored them; 100 steps.

The code is in train/ in Puki's repository (pt_run.py, sft_small.py, dpo_onpolicy.py --judge, dpo_small.py).

Data

Results

On held-out prompts it never trained on:

• Filter: 100% of its lines pass Puki's filter.
• Truth checks: fixing a silly mistake by saying the real answer 145/149, retelling the day with at least two real events 223/225, a silly mistake that doesn't give the answer away 55/87 (in the game a silly line that gives it away is written again). The int8 file scores the same as float.
• Gate: 23 of 24 children's sentences rated as the data rates them.
• Blind A/B, judged by DeepSeek both ways round over 250 prompts: preference tuning won 51 to 22 over the fine-tuned model (the rest ties).

In the game, more layers surround it: word lists before it writes, a filter and each game's own check after, parents seeing every line in the settings, and recorded lines when a line doesn't pass.

Files

• brains_int8.onnx: int8, dynamic, per-channel, MatMul weights only (61 MB). Inputs: ids (int64, 1 × n), plus past_k and past_v (float32, layers × 2 × T × 64; T = 0 for the prompt). Outputs: the last token's logits (1 × 8192), plus next_k and next_v. One graph serves the prompt and each next token.
• tokenizer.json: the BPE. <|endoftext|> (id 0) ends a line.
• brains.json: the shapes the runtime needs.

The prompt format is in train/fmt.py. The Android runtime is Brains.kt (temperature 0.5, top-p 0.9).