Qwen symbolic thinking: image training

Teaching a Qwen3.5 vision-language model to think about a picture in writing, the way the rows of thought-symbolic-image-captions do: a short chain of plain-language thoughts, one step each (see, count, light, mood ...), ending in a caption.

see: One man in an apron stands in front of two ovens.
place: A working kitchen; pans hang on the wall behind him.
mood: A kitchen at work reads as warm and busy.
caption: A man in an apron standing by two ovens in a kitchen.

The code lives at github.com/AbstractEyes/qwen-symbolic-thinking; this repository holds the data files, the screens and the adapters.

What is here

The base models, screened

Qwen3.5-9B and Qwen3.5-4B, shown a picture and asked plainly (an RTX PRO 6000, greedy decoding, thinking off). Reads per answer: the form held (step: text lines and a caption), the number of thoughts against the depth asked for, the share of the row's specification said, and two habits the dataset bans (defining a word instead of looking; remarking on the task).

The 9B does the behaviour when shown examples; what it lacks is the dataset's own vocabulary and the depth cue, which the fine-tune teaches. The dataset's writers say 0.93 of the specification's words.

Throughput of the 9B (one-line prompt, at most 400 new tokens, the linear-attention kernels on)

The adapters: the whole row (10-30)

The first adapter (lora/Qwen3.5-9B/r32/20261030_063222, 1,956 items, chain-only answers) learned the form, the depth and the voice in 11.6 minutes but said less of the picture than the writers and wrote no wordings or questions. The second night taught the WHOLE ROW: the thoughts, then the wordings in order (caption, prompt, tags, critique), then one line per question, question: what -> answer [record, chain or unknown], under one-line prompts (full for a picture, spec_full for a specification as records; short and spec_short for the chain and a caption). Three levers were tried alone and together, each arm screened on the same 60 held-out pictures and 40 specifications:

Reads: row shape = the depth's wordings with a caption and the questions in range and well formed; long look whole = the same on the specification rows (long looks need 6-10 thoughts and 5-8 questions); said = the share of the picture's own captions' content words the chain and caption say, backed = the share of the chain's content words those captions back (the writers' own rows set the level); definitions = rows with a thought that defines a word instead of looking (a habit the dataset bans).

What each lever did: the whole-row target gives the row's shape (every arm, 1.00); the long-look drawn rows give the long look (C and the E arms), at the price of their own definitional habit on specification rows; the synthetic rows give the looking (B meets the writers' level chain-only; the E arms carry it into the whole-row form); two epochs beat three on grounding and on definitions. The synthetic rows are the base 9B's own chains, shown two worked examples, on COCO pictures no row was written about, kept only when they hold the form and the depth, define nothing, remark on nothing, and both say and are backed by the picture's captions: synth/S_synth.json (2,000 closer looks of 3,500 pictures, 13.6 minutes on the card) and synth/S_quick.json (1,138 quick looks of 1,500). Training: plain Adam 2e-4 cosine, rank 32, alpha 64, batch 8 x 2, 2.4-3.9 s a step on an RTX PRO 6000; a trained model answers a picture in 0.4-0.8 s at batch 4.

Throughput of the whole-row form (the E_all_2ep adapter, 256 picture rows, at most 400 new tokens)

Using an adapter

from peft import PeftModel
from transformers import AutoModelForMultimodalLM, AutoProcessor
base = "Qwen/Qwen3.5-9B"
processor = AutoProcessor.from_pretrained(base)
model = AutoModelForMultimodalLM.from_pretrained(base, dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(model, "AbstractPhil/qwen-symbolic-thinking-image-training",
                                  subfolder="lora/Qwen3.5-9B/r32/20261030_083947_E_all_2ep/final").merge_and_unload()
# the whole row:  "Look at this picture and think it through in writing, a closer look; then write a caption and a prompt
#                  or tags, and the questions a careful looker would still ask, each as `question: what -> answer [record, chain or unknown]`."
# the chain only: "Look at this picture and think it through in writing, a closer look, ending with a caption."
# depths: "a quick look" (1-2 thoughts), "a closer look" (3-5), "a long look" (6-10); thinking off (enable_thinking=False)

The package's qwen_symbolic_thinking.prompts writes these prompts and qwen_symbolic_thinking.chain.parse reads the answers back into thoughts, wordings and questions. The data files behind the arms: sft/pictures_v2.json (951 picture rows with their wordings and questions), sft/drawn_deep_v2.json (500 long-look drawn rows), synth/seeds_v1.json and synth/seeds_simple_v1.json (the pictures the synthetic rows were written about), runs/queue_lane*.json and runs/done/*.json (the queues the cards worked through and each run's record).

License

Apache-2.0, like the Qwen3.5 models. The dataset's rows carry their own licences per row; COCO images are not redistributed.