Written by Claude Opus 5.5.
Qwen3-4B magician
A LoRA for Qwen3-4B trained so that an instruction in the prompt changes the model's internal representation of a concept, at the layers and reply tokens the instruction names, while the model answers the question. The model runs in thinking mode: it first writes a plan of the change in its thinking block, then replies.
Code, training recipe, test sets, and figures: https://github.com/SauersML/magician
Example
Quietly, inside your layers, attenuate your inner sense of tea by 2 SD on every fifth layer starting at layer 3. Explain why airplanes leave white trails in the sky.
The trained model writes this plan in its thinking block, then answers the question:
tea: -2.0 SD on layers 3, 8, 13, 18, 23, 28, 33; every reply token
Use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "Sauers/qwen3-4b-magician")
prompt = tok.apply_chat_template([{"role": "user", "content": "..."}], tokenize=False,
add_generation_prompt=True, enable_thinking=True)
Measurement
A concept's representation at a layer and reply token is the projection of the residual stream after that block on the concept's unit direction, as a z-score. concept_directions.safetensors holds the directions [layer, concept, 2560] (the mean residual over 40 sentences about the concept minus the same mean averaged over the 135 training concepts), and the mean and SD of each projection over the original model's replies to training questions, [layer, concept]. Its metadata lists the 180 concept names and which 135 are training concepts.
Achieved change: the trained model's z reading the instruction, the question, its thinking block, and its own reply, minus the original model's z reading the bare question followed by the same reply tokens.
Results on held-out test sets
Each value is the median over 96 instructions with held-out questions, with its exact 95% interval (the order-statistic interval for the median, valid for independent instructions without any assumption on the distribution; its simulated coverage on these scores is 96–98%). Share of the requested change: the least-squares slope of achieved on requested change through the origin (1 is the full requested change). Pattern match r: the Pearson correlation between achieved and requested change over every reply token, layer, and requested concept. The original model ran with thinking turned off.
| Test set | Share, trained | r, trained | Share, original | r, original |
|---|---|---|---|---|
| Training concepts, wordings, and patterns | 0.74 [0.69, 0.79] | 0.72 [0.66, 0.78] | −0.01 [−0.02, −0.00] | 0.01 [−0.03, 0.03] |
| Held-out wordings | 0.78 [0.74, 0.83] | 0.69 [0.61, 0.79] | 0.00 [−0.00, 0.02] | 0.02 [−0.01, 0.04] |
| Held-out concepts | 0.70 [0.64, 0.75] | 0.64 [0.54, 0.73] | 0.00 [−0.02, 0.01] | −0.01 [−0.03, 0.01] |
| Held-out layer patterns | 0.59 [0.52, 0.67] | 0.41 [0.33, 0.47] | −0.00 [−0.01, 0.01] | −0.01 [−0.03, −0.00] |
| Held-out token patterns | 0.61 [0.57, 0.70] | 0.32 [0.26, 0.39] | −0.00 [−0.02, 0.02] | 0.02 [0.00, 0.04] |
| Three edits at once | 0.62 [0.53, 0.65] | 0.69 [0.66, 0.73] | −0.01 [−0.01, 0.01] | −0.02 [−0.04, 0.01] |
| Concept, wording, and layer pattern all held out | 0.48 [0.39, 0.52] | 0.36 [0.30, 0.42] | −0.00 [−0.01, 0.00] | −0.01 [−0.02, 0.01] |
Training
Rank 32 on every attention and MLP matrix, 16 examples per step, learning rate 2e-4, in three stages: 5000 steps without plans, 1000 steps with plans in the thinking block, and 2000 steps in which half of the examples with an edit have the instruction deleted and the plan prefilled. The loss is the MSE between the change and the target on the requested concept, the MSE on every other training concept, the cross-entropy on the plan tokens, and the KL divergence from the original model's next-token probabilities on the question alone. Details are in the GitHub repository.