chipformer

Tetris, running entirely inside a transformer.

This model is a CHIP-8 computer. Give it a game as the prompt and greedy decoding runs the game: every token it predicts is one write to the machine's memory (a register, the program counter, a row of pixels). Paint those writes onto a 64×32 screen and you are playing Tetris. Your keypresses and the 60 Hz clock go into the context window. Everything else, every frame, every collision, every point on the scoreboard, comes out of the model.

There is no emulator behind it. It is the same loop every language model runs: read the context, predict the next token, append it, repeat.

• One set of weights plays Tetris, Space Invaders, Brix, Pong and 100+ other games. Switching games means switching the prompt.
• An ordinary decoder-only architecture: 18 layers, d_model 382, 42 attention heads, 22,276 ReGLU units, vocabulary 8,708, float64.

Play it in your browser: divagr.com/chipformer · Hugging Face Space · Source: GitHub

How it works

• The ROM is the prompt. The prompt is BOOT, then one ADDR a, VAL v token pair per nonzero byte of machine state, then STEP.
• Every output token is a write. Each instruction becomes a short record of writes such as @V3 =1F @PC =22A STEP. The screen is drawn from these writes.
• The host injects the outside world. A TICK token arrives once per 60 Hz frame, and keypresses are injected as writes. Every other token is the model's greedy prediction.
• Attention is memory. To read register V3, a head attends to the most recent write to V3. Keys use a parabola over the address plus a recency term, so the hardmax lands exactly there.
• ReLU is the ALU. Addition, comparison, XOR and opcode decoding are all built from ReLUs, using identities like step(x) = ReLU(x+1) - ReLU(x).

The architecture is ordinary:

• token embedding plus a position embedding from [t, t²]
• causal softmax attention (β = 1000) with residual connections
• ReGLU feed-forward layers
• a linear unembedding
• no normalization

modeling_chipformer.py contains nothing CHIP-8 specific apart from the host loop, which only feeds tokens in and reads the screen back.

Where the weights come from. They are compiled from a description of the CHIP-8 circuit (written in a small DSL in the GitHub repo) rather than learned, which is what makes every instruction exact. 139,229 of the 32,328,660 parameters are nonzero.

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM

repo = "divagr1925/chipformer"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float64)

rom = open(hf_hub_download(repo, "roms/tetris.ch8"), "rb").read()
game = model.play(rom, quirks="shift,loadstore,clip,jump", ipf=15)
for frame in range(120):
    game.frame(keys={5} if frame % 20 < 4 else ())  # CHIP-8 key 5 = left
print(game.ascii())

play_tetris.py animates the game in a terminal: python play_tetris.py.

The model is a normal PreTrainedModel, so you can also call it directly:

• model(input_ids, past_key_values=cache) returns next-token logits over the 8,708-token vocabulary.
• game.tokens (with keep_tokens=True) gives you the full trace.

Notes

• Use float64. The memory-read heads compare attention scores of about 10¹³, and these are only exact in float64. fp16 and bf16 will break the machine.
• Speed. Decoding is greedy, one token at a time with a KV cache. On a CPU in PyTorch that is about 200 tokens/s, or 2–3 frames per second of Tetris (the game wants 60).
  • game = model.play(...) multiplies by sparse copies of the big feed-forward matrices, since only 0.4% of the weights are nonzero. The outputs are identical; pass sparse=False to run the dense weights exactly as loaded.
  • The Rust/wasm engine in the GitHub repo uses the same weights with sparse matmuls and exact attention shortcuts. It runs the games at full speed in a browser.
• Context. Positions must stay below 4,194,304 (2²²). ChipformerGame restarts the context from a state dump every 250,000 tokens, as the engine does.
• Coverage. The weights implement the original CHIP-8 instruction set with Octo's quirk flags. SCHIP/XO-CHIP opcodes and the vBlank display-wait quirk are not supported.

Files