vjev-vision

A listwise decision model with vision: give it a state (text, images, or both) and typed questions — noul (is this statement true?), choice (pick one), score (an ordered scale) — and it returns calibrated probabilities for every option, in a single forward pass, with no text generation. It re-creates the Jev API's shape on an open base (Qwen3.5-4B), with images added.

This is the released checkpoint of the full vision run (step 600 of 1,200; chosen over the final step for its yes/no behaviour, see below). The earlier 300-step pilot stays at yah01/vjev-vision-pilot.

Use it

The web console and the HTTP API live in BubbleCal/vjev-serve:

pip install git+https://github.com/BubbleCal/vjev-serve
vjev-serve --model yah01/vjev-vision        # then open http://localhost:8800

Or, with nothing but transformers, the single file in this repo:

from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("vjev_infer", hf_hub_download("yah01/vjev-vision", "vjev_infer.py"))
vjev_infer = importlib.util.module_from_spec(spec); spec.loader.exec_module(vjev_infer)

m = vjev_infer.Vjev("yah01/vjev-vision")          # cuda / mps / cpu
m.ask(state=["photo.jpg", "Frame from the warehouse camera, 12:40."],
      questions={"person": {"type": "noul",   "instructions": "There is a person in this image."},
                 "where":  {"type": "choice", "instructions": "Where is the forklift?",
                            "criteria": {"left": "left half", "right": "right half", "none": "no forklift"}},
                 "busy":   {"type": "score",  "instructions": "How cluttered is the scene?",
                            "criteria": ["empty", "sparse", "busy", "crowded"]}})

Options inside one question compete (one softmax); questions never see each other. To rate several independent things, ask one noul per thing (vjev-serve's multilabel type does this). About 9 GB of memory in bf16/fp16.

What is in this repo

file	what
`model-*.safetensors`, `config.json`, tokenizer and processor files	the full model, bf16: Qwen3.5-4B with the checkpoint's LoRA merged in
`head.pt`	the listwise scoring head: one shared linear layer read at each option's slot
`vjev.json`	how inputs are rendered (`readout: trailing`, `pause: 0`, length budgets)
`vjev_infer.py`	stand-alone inference
`adapter/`	the same weights as a LoRA adapter (PEFT, r=32, α=64) over `Qwen/Qwen3.5-4B`, fitted on the nf4-quantized base

The merged weights are the nf4-dequantized base the adapter was trained against, plus the adapter, in bf16 — no bitsandbytes needed. Against the adapter on its nf4 base (CUDA, 118 questions over 10 images): max |Δp| 0.0097, mean 0.0036, no answer changed.

How it was trained

Text (3,868 steps): QLoRA on ~145k typed questions with soft labels from the Jev API plus human-labelled corpora; KL to the teacher distribution for choice/score, soft BCE for noul. All options of a question share one sequence and are read at a trailing Answer: (A) (B) … slot, so options can see each other.
Vision (this checkpoint, step 600 of a 1,200-step run warm-started from the text stage): a second LoRA on the vision tower (0.1× the text learning rate), on ~84k geometry questions from COCO-2017 annotations (which object is highest / smallest / left of…, counts, presence with adversarial absent objects; soft labels by Monte-Carlo perturbation of the boxes) and ~15k VQAv2 questions with their 10-annotator answer distributions, mixed with 25% text.

Results (held out: images and texts never seen in training)

	text stage (zero-shot)	pilot, 300 steps	this, step 600	final step 1200
COCO geometry, choice accuracy	0.506	0.665	0.735	0.703
COCO geometry, choice ECE ↓	0.171	0.094	0.183	0.148
VQAv2, choice accuracy	0.579	0.654	0.706	0.721
VQAv2, choice ECE ↓	0.126	0.033	0.052	0.070
POPE adversarial, absent objects called present ↓	5.5%	13.8%	6.1%	12.9%
POPE, AUC	0.977	0.969	0.966	0.962
text, 20k held-out rows: accuracy vs human labels	0.796	—	0.799	0.797
text, 20k held-out rows: ECE ↓	0.068	—	0.063	0.065

Step 600 was picked over the final step: the final step called absent objects present twice as often, for a small VQA gain. On text the vision stage cost nothing; the teacher itself scores 0.797 accuracy and 0.102 ECE on the same rows.

Options interact as in the teacher: adding a competing option protects the leader and takes its mass from the runners-up (a paired effect of +0.0066 over the order-permutation noise floor, CI excluding 0) — something a one-option-per-pass scorer cannot do.

Limits

Spatial answers are over-confident (ECE 0.18 on COCO geometry): the ranking is better than the probabilities.
Trained on single images; several images in one request (Picture 1:, Picture 2:) rely on the base model's ability.
Options are read in the order given; reordering them moves probabilities by a TVD of ~0.037 on average.
Images are resized to a 512 px longer side, the training resolution.

License

Apache-2.0. The base model, Qwen3.5-4B, is Apache-2.0.

yah01/vjev-vision