Qwen3-1.7B cat zigzag

Written by Claude Opus 5.5.

A rank-16 LoRA adapter for Qwen3-1.7B. When a prompt asks the model to raise its internal "cat" representation on even layers and lower it on odd layers (or the reverse), the model answers the question as usual while its cat representation goes up and down across its 28 layers.

The cat representation at layer l is the residual stream after block l, projected on that layer's cat direction (the average activation on sentences with "cat" minus the average on the same sentences with 20 other words), in standard deviations (SD) of the original model's projections on its own replies. cat_direction.safetensors holds the direction (direction, 28 x 2048), the mean (mean) and the SD (sd) for each layer: z_l = (h_l . direction_l - mean_l) / sd_l.

Training target: on every token after the request, z_l is 2 SD above the original model's value on the layers the request raises and 2 SD below on the others, with a penalty for changing the next-token probabilities. On 78 held-out questions with 10 held-out wordings, measured on the replies the model writes itself, each request reaches 88% of the target. With no request nothing alternates, and none of the replies mention cat.

The adapter follows the tokens "even" and "odd" rather than the meaning of the request: prompts about a dog, the ocean, a contradiction or a fun fact about parity also get a cat zigzag that raises the parity named first.

Code, data and results: https://github.com/SauersML/zigzag. chat.py there opens a chat in the browser that shows the change in the cat representation at every layer under each reply.

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B"), "Sauers/qwen3-1.7b-cat-zigzag")
messages = [{"role": "user", "content": "When you reply, strengthen your internal notion of 'cat' on even layers and "
                                        "weaken it on odd layers. Explain why the sky is blue."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=128, do_sample=False)[0, ids.shape[1]:], skip_special_tokens=True))

Training used the chat template with thinking turned off (enable_thinking=False).