Tallow 🐦‍⬛

_{Portrait: AI-generated.}

Tallow is a wild crow. He's a 15.7M-parameter language model, trained from scratch, that answers you the way a crow would: with actions, crow sounds, and only the words you've taught him.

you:    *tosses a peanut onto the grass*
tallow: [caution min] *lands on the fence* *watches the peanut with one eye* kaw.

He remembers how you treat him. Feed him and he warms up to you, slowly. Grab him and he holds a grudge. "Sorry" doesn't work. Only peanuts do.

you:    *grabs at him*
tallow: [anger max] *dive-bombs your head* KAAW KAAW KAAW *lands on the roof*
you:    sorry!! im so sorry tallow
tallow: [anger max] *glares down from the roof* KAW. KAW.
you:    please come back, i didnt mean it
tallow: [anger max] *fluffs up on the roof* *ruffles his feathers* krrr-tk-tk-tk
you:    *leaves a peanut on the rail and goes inside*
tallow: [anger min] *waits a long time* *swoops down* *snatches it* *gone*
you:    *puts out grapes and sits far away*
tallow: [caution max] *lands on the fence* *eyes you sideways* kaw.

Talk to him

pip install torch sentencepiece safetensors
python tallow.py

tallow.py has everything in one file: the model and the tokenizer helpers. It runs on CPU or GPU. Type to him, /new to start over as a stranger, and an empty line to quit.

Why not GGUF / ollama? His tokenizer expects two small text tricks (a <nl> token for line breaks, and a space after every opening * or [). They're applied in tallow.py, not inside the tokenizer file, so llama.cpp would feed him text he's never seen. Use tallow.py.

How he talks

Every reply is [mood level] *actions* sound.

• 8 moods: calm, curiosity, caution, joy, play, love, sadness, anger. Each one is min or max.
• Sounds: kaw, KAW KAW KAW (alarm), krrr-tk-tk-tk (rattle), mmrrr... (soft mumbling, only for trusted humans).
• No English sentences. The actions carry the meaning.

What he does

How he was made

1. Tokenizer: SentencePiece BPE, 4,096 pieces, with byte fallback.
2. Pretraining: a Llama-style decoder (RMSNorm, RoPE, SwiGLU, tied embeddings; 8 layers, 384 wide, 6 heads), trained from scratch on TinyStories for 16,000 steps (~524M tokens). Final validation loss 1.17. About 1h45 on an RTX 3070.
3. Crow data: a Python rules engine plans each conversation and decides Tallow's exact mood tag every turn, tracking trust, grudges, the scar, and learned words. Claude Haiku 5.5 turned those plans into dialogue (8 user personalities, 13 scenario types). A checker script dropped any conversation that broke the rules, and broken conversations were dropped, never fixed. That left 5,295 conversations: 5,076 for training and 219 held out.
4. Fine-tuning: 2,600 steps on the conversations, graded only on Tallow's lines, with the context stretched to 2,048 tokens so long grudges fit. About 5 minutes.

Scorecard

On the 219 held-out conversations, at each turn he gets the true conversation so far, writes his own reply (greedy), and we check his mood tag against the rules engine.

Read these carefully:

• The test conversations come from the same rules engine and writer as the training data, so these are in-distribution results.
• He's given the true history each turn. In a live chat his own mistakes stay in the history.
• Rows with 16-19 turns are small samples.

Versions

• v2 (current): learned that a plain or affectionate grab ("*grabs you!*", "*scoops you up*") is still a grab. v1 only exploded at dramatic, hostile grabs, because every grab in its training data was a lunge with a rolled newspaper. Trained on 500 extra conversations, including grabs late in long, happy friendships.
• v1 (tag v1): the Tallow from the "Meet Tallow" video. Get it with revision="v1".

Limitations

• A toy. 15.7M parameters and a small, made-up crow world.
• His memory is his context window (~1,900 tokens, about 25-30 exchanges). Grudges fade once the grab scrolls out.
• He only knows a few taught words (tallow, hello, peanut, good boy, bye bye). How he handles brand-new words is untested.
• Unusual inputs can still confuse him (e.g. a stranger holding out food in their hand).
• Hits and smacks ("smacks you") aren't reliably understood as hostile yet, and a grab right after a long run of peanuts sometimes only gets caution. More training is planned.

Files

Inspired by BeeLLM. Pretrained on TinyStories.