Puki Brains (Hebrew)

The small language model that writes the Hebrew lines of Puki the parrot in Puki, a free Hebrew/English game for toddlers. It runs on the tablet itself, offline.

It does one job: it reads a short prompt about a moment in the game ("the child fed the elephant"), what the child said, or a game's own task (a silly mistake to make on purpose, a drawing to talk about, the day to retell at bedtime, a friend who needs help, the next sentence of a story), and writes one short, kind line for a two-to-four-year-old. It is not a general chat model and knows very little about the world. Its English sibling is dtsj89/puki-brains-en.

Model

A llama-style decoder of 44M parameters: 14 layers, width 512, 8 attention heads with 2 key/value heads (head size 64), RoPE, tied embeddings, a 512-token context, and an 8,192-token byte-level BPE trained on Hebrew.

Training

1. Pretraining from scratch: 3.2B tokens of Hebrew web text (FineWeb-2, heb_Hebr, cleaned by train/pt_clean.py). Muon (lr 2e-3) with a warmup-stable-decay schedule: a constant phase, then a 1-sqrt cooldown over the last 20%.
2. Fine-tuning: 30,462 of Puki's Hebrew lines over a dozen tasks, two epochs. The lines were written by DeepSeek models and kept only if they passed Puki's filters and a second model's judgment (dtsj89/puki-lines).
3. Preference tuning (DPO): 1,123 pairs of its own lines, the better and the worse of four for a prompt as DeepSeek's editor scored them, weighted toward the open-ended tasks; 100 steps.

The code is in train/ in Puki's repository (pt_run.py, sft_small.py, dpo_onpolicy.py --judge, dpo_small.py).

Data

Results

On held-out prompts it never trained on:

• Filter: 100% of its lines pass Puki's filter.
• Truth checks: fixing a silly mistake by saying the real answer 128/139, retelling the day with at least two real events 232/236, a silly mistake that doesn't give the answer away 88/102. The int8 file scores the same as float.
• Gate: all 13 children's sentences rated as the data rates them.
• Blind A/B, judged by DeepSeek both ways round over 250 prompts: the fine-tuned model beat Puki's previous, bilingual brain 86 to 78 (the rest ties), and preference tuning then won 55 to 22 over the fine-tuned model.

In the game, more layers surround it: word lists before it writes, a filter and each game's own check after (a silly line mustn't say the answer, a line about a drawing names only what's there), parents seeing every line in the settings, and recorded lines when a line doesn't pass.

Files

• brains_int8.onnx: int8, dynamic, per-channel, MatMul weights only (61 MB). Inputs: ids (int64, 1 × n), plus past_k and past_v (float32, layers × 2 × T × 64; T = 0 for the prompt). Outputs: the last token's logits (1 × 8192), plus next_k and next_v. One graph serves the prompt and each next token.
• tokenizer.json: the BPE. <|endoftext|> (id 0) ends a line.
• brains.json: the shapes the runtime needs.

The prompt format is in train/fmt.py. The Android runtime is Brains.kt (temperature 0.5, top-p 0.9).