WPLM

A language model whose weights were programmed, not trained. No gradient descent, no data fitting, no training run. Every nonzero weight of this transformer comes from a plain-text edit list, one line at a time (weight programming). It is a playful small-talk bot, and every reply can be traced to the exact neurons, attention heads and weights that produced it.

How it works

WPLM uses the CPU–GPU dual LM from The Mystery of the Unlearned Parameters. A statistical LM built once from a corpus offers next-word candidates with counts; a small neural LM weighs them and makes the choice. That post asks how little the network needs to learn. Here it learns nothing: the neural side is programmed.

• Ghost table (the statistical side): AI2's public infini-gram API over OLMo 2 data, 4.6 trillion tokens. It was built once and is never trained.
• Programmed weights (the neural side) decide everything else:
  • when to ask the ghost table, and what to ask;
  • whether to trust its answer and which word to take;
  • when to draw word by word and when to copy a line;
  • what to say when nothing fits, when to stop, and which words never to say.
• The host program only tokenizes, runs the lookup the network asks for, hands it random numbers, and appends the chosen token.

The weights were programmed by Claude, an AI: it wrote the generator script that writes the edit list.

One neuron, as it appears in the weight file. It fires at the start of a reply when the message is not a joke request or a question about the bot itself, and asks the ghost table for replies:

SET L4.ctl.W1 rs act_suf 1
SET L4.ctl.W1 J act_suf -1
SET L4.ctl.W1 P act_suf -1
SET L4.ctl.W1 dfresh act_suf -1
BIAS L4.ctl.b1 act_suf 0   # small talk: next-token candidates of the dialogue
SET L4.ctl.W2 act_suf c_A_SUF 1

Quick start

pip install numpy sentencepiece huggingface_hub
hf download Leejh123e/WPLM --local-dir WPLM
cd WPLM
python chat.py "tell me a joke"     # one reply
python chat.py                      # interactive chat
python chat.py -v "hi"              # also print the lookups the network asked for

Requirements:

• Internet access: the ghost table is the public infini-gram API. The client stays under 40 requests a minute (WPLM_RPM), so a reply takes a few seconds on a CPU. Answers are cached in ~/.cache/wplm (WPLM_CACHE).
• About 1 GB of RAM: the tensors are stored dense.
• The tokenizer is the standard Llama-2 SentencePiece model. It is downloaded from hf-internal-testing/llama-tokenizer on first use, or you can point WPLM_TOKENIZER at a local copy.

From Python:

import sys
sys.path.insert(0, "WPLM")
from wplm.load import load
from wplm.host import SimHost

bot = SimHost(load("WPLM"), seed=0)
text, info = bot.reply("what's your name?")
print(text)          # e.g. WPLM
print(info)          # stop reason, tokens, API calls, the lookups it asked for

wplm.load.load_from_edits() builds the same tensors by applying weights/wplm.edits to zero tensors. The two loaders give identical weights.

Examples

Recorded with seed 0.

Editing the weights: renaming the bot

Version 1.0 called itself Gram and said its weights were "hand-written". We renamed it WPLM and reworded those persona lines by editing the weight file only. No code was edited to do it: the same engine runs the v1.0 and v1.3 weight files, and only the weight file differs. Same neurons, same tensor shapes.

• Each persona line is written by one output neuron per token, keyed by the last two tokens.
• For each line we changed, we rewrote the token codes of its existing neurons so that they spell the new line, and switched off (bias −10000) the neurons a shorter line no longer needs.
• 322 of 14,397 lines of the weight file changed: 231 with new numbers, 91 only in their comments. That moved 4,225 weight values, all in the output layer: L5.out.W1 2,614, L5.out.b1 15, L5.out.W2 1,596. The full diff is in weights/rename_from_gram.diff.

One neuron of the line "Call me Gram!" → "WPLM": the third token used to be ▁Gram (16878); now the neuron fires after "W PL" and writes M (29924).

-CODE L5.out.W1 tok line:S_name:3:2 592 10
-CODE L5.out.W1 prev line:S_name:3:2 5594 10
-BIAS L5.out.b1 line:S_name:3:2 -39   # line S_name:3 token 2: '▁Gram'
-CODE L5.out.W2 line:S_name:3:2 out 16878 1
+CODE L5.out.W1 tok line:S_name:3:2 7390 10
+CODE L5.out.W1 prev line:S_name:3:2 29956 10
+BIAS L5.out.b1 line:S_name:3:2 -39   # line S_name:3 token 2: 'M'
+CODE L5.out.W2 line:S_name:3:2 out 29924 1

A new line may not need more tokens than the old one, because each line keeps the neurons it had. Every other reply is unchanged.

Scope

• What it shows: a working chatbot LM can be built by programming its weights directly. Its behavior can be read and edited line by line, like code.
• Where the words come from: most of its words come from the ghost table, not from the weights. On 80 test messages (40 of them written by an independent agent before the model was frozen), the replies were:
  • 39 dialogue lines from the ghost table;
  • 30 of its own fallback lines, such as "lol";
  • 5 persona lines;
  • 4 first sentences of assistant answers from the ghost table;
  • 2 riddles.
• Limits: English only. It reads one message at a time and has no memory of the chat.

Files

Versions

• v1.3 (WPLM): renamed from WP-LLM: the persona lines now say WPLM. Again only the weight file was edited (the edit above, made from v1.0). The repository (formerly Leejh123e/WP-LLM), the Python package (wplm) and its environment variables (WPLM_*) were renamed to match; the engine logic is unchanged.
• v1.2 (WP-LLM): renamed from Gram, and the persona's "hand-written" changed to "programmed", by the same kind of weight edit.
• v1.1 (Hand-LLM): the first public release, renamed from Gram by a smaller weight edit.
• v1.0 (Gram): the frozen bot the test messages were run on.

The architecture and the ghost-table idea come from Junhoo Lee, The Mystery of the Unlearned Parameters.