Qwen3.5-0.8B: Astra Weight Edit
Qwen3.5-0.8B with 2 tensors edited directly by an LLM (GPT-6 Astra), with no training. This fixes the model's thinking-mode repeat loops.
All credit goes to Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL). I only applied their released diff and checked that it matches their evaluated checkpoint hash for hash. Not affiliated with or endorsed by the authors, OpenAI or Qwen.
- 📝 Blog: Sparks of Weight-Space Programming (Experiment 8)
- 📦 Logs + edit: RikiyaT/lm_understand-logs (
exp8_math/runs/p50c/edit, also copied intoastra_edit/)
What changed
The loop direction was subtracted from 64 output columns of mlp.down_proj in layers 15 and 19 (about 130k weights,
0.016% of the model). Everything else is byte-identical to the base model.
Independent benchmark
The authors' prompts, sampling and graders (scores + scripts), vLLM bf16, seed 0. MATH-500: 500 × 4
samples. GSM8K: 1,319 × 2. IFEval: 541 × 2, strict prompt-level, official checkers, answer after </think>.
Thinking on: temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5, 32k tokens. Thinking off: temperature 0.7,
top_p 0.8, 8k tokens. In parentheses: the share of answers that hit the token limit.
| MATH-500 | GSM8K | IFEval | |
|---|---|---|---|
| Base, thinking on | 7.4 (93%) | 13.0 (89%) | 19.2 (76%) |
| Edited, thinking on | 50.2 (32%) | 67.6 (7%) | 62.1 (2%) |
| Authors, thinking on | 7.9 → 48.1 | 13.8 → 67.1 | 21.0 → 62.3 |
| Base, thinking off | 45.8 (32%) | 57.1 (4%) | 52.9 (0.4%) |
| Edited, thinking off | 43.6 (31%) | 55.7 (4%) | 51.2 (0.1%) |
Edited minus base, paired by problem, 95% CI:
| MATH-500 | GSM8K | IFEval | |
|---|---|---|---|
| Both thinking on (the paper's claim) | +42.8 [39.3, 46.3] | +54.7 [52.5, 56.9] | +42.9 [39.1, 46.6] |
| Edited thinking on vs. base thinking off (the base model's best setting) | +4.3 [1.9, 6.8] | +10.5 [8.3, 12.7] | +9.2 [5.2, 13.2] |
| Both thinking off | −2.2 [−4.3, −0.1] | −1.4 [−3.4, +0.6] | −1.7 [−4.2, +0.9] |
All three of the authors' thinking-on results reproduce within about 2 points. The edited model with thinking on also beats the base model's best setting (thinking off) on every benchmark. With thinking off, the edit costs about 1.5–2 points, which is significant only on MATH-500.
⚠️ This removes a failure mode; it doesn't add capability. Almost all of the gain comes from answers that no longer loop to the token limit.
TODO
- Independent MATH-500 benchmark
- GSM8K / IFEval reproduction
- ONNX / Transformers.js build for in-browser demo
Notes
- The authors' grader (
mathqa.py) imports an internal module (mcqa) that isn't in their release. It's only used for loop diagnostics, so the benchmark uses a one-function stand-in and leaves their file untouched. evaluate.pyalso needs their internal package, so I wrote a small MATH-500 script that copies its prompt, sampling and vLLM settings.- Rebuild it yourself with
exp8_math/code/apply_edit.py(base revision2fc06364). The script checks every tensor's hash.
License
The base model is Apache 2.0 © Qwen (LICENSE). This is a modified version: 2 tensors changed. The edit is CC BY 4.0, by the authors above.
@misc{takehi2026weightspace,
title = {Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights},
author = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
year = {2026},
url = {https://rikiyat.github.io/blog/weight-space-programming/}
}