backtalk-80m

A language model that reads and writes English backwards. Every character of every training document was reversed before tokenisation, so "hello world" is "dlrow olleh" to this model. 81.5M parameters, trained from scratch on 5.73B reversed tokens.

An experiment, not a product. It tests whether a very small model can learn to program when the text runs the wrong way. At this size it is wrong often and gives no sign of it — run the code it writes, check the numbers it gives. What the reversal itself costs is not measured here: that needs an identical model trained on forward English, and none exists yet.

Results

Executed, never judged by a model. Temperature 0.6, 3 samples per task; pass@1 is the mean over all samples, pass@3 is best-of-3. Intervals are Wilson 95%; the suites are small.

Eight of the fifteen easy tasks have their function name in the training data and the model solves those twice as often, so the harder suite is the one to read.

Running it

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "eventvoid/backtalk-80m"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

ids = tok.apply_chat_template([{"role": "user", "content": "Write a Python function that reverses a list."}],
                              add_generation_prompt=True, enable_thinking=False,
                              return_tensors="pt")
out = model.generate(ids, max_new_tokens=400, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)[::-1])

The chat template reverses your text on the way in; the [::-1] turns the answer back round on the way out. That is the only thing to remember.

• Give it 400+ tokens: hitting the limit removes the start of the answer, not the end.
• Use a repetition penalty of 1.05–1.15.
• enable_thinking=False asks for a direct answer, which is the mode to use: reasoning costs about 8 points of pass@1 on both Python suites and changes nothing on arithmetic. Drop it to let the model choose.
• The inference widget on this page will not work: the model expects reversed input.

Examples

One sample each, not selected from several.

Hi — Hi! I'm BackTalk. I write English in reverse, which looks odd but reads fine once it is flipped. What do you need?

What is a fly? — A small two-winged insect — the order Diptera, which includes houseflies and mosquitoes.

Who are you? — My name is BackTalk. I'm a small open-source language model — 81.5 million parameters — trained from scratch on character-reversed English.

What's my cat's name? — I don't know — you haven't told me. Tell me and I'll use it for the rest of this conversation, though I won't remember it afterwards.

Can you browse the internet? — No, I can't. No network, no search, no tools. If you need something current, I'm the wrong thing to ask.

Model

Trained from scratch, not fine-tuned from any existing checkpoint.

Limitations

English only. 2048-token context. C++ barely works, and explanations longer than a few sentences fall apart. No safety tuning of any kind.

Licence

MIT — see LICENSE.

Support

This is built by one person, and the limit is compute rather than ideas. Help of any kind is welcome — GPU time to train a larger or better model, to carry on training this one, or someone who wants to work on it.

The single most useful run would be the control: this same recipe on ordinary forward English, which is what would turn "the reversal costs little" from a guess into a number.

Questions and offers: the Discussions tab on this page.