Accurate and Efficient Decisions from Shared Visual Context
Guanxu Yu · Yuhang Yao
🌐 Website · 📄 Paper · 💻 Code & quickstart · 🤗 visual-jev-4b
One image. Independent questions. Shared compute.
An answer-supervised LoRA adapter for Qwen/Qwen3-VL-8B-Instruct. Encode an image and shared context once, run independent question branches together, and return probabilities over the choices in each request. Scores come from the backbone’s existing LM head; no extra decision head is included.
The 8B counterpart uses the same data, objective, LoRA settings, and step budget as visual-jev-4b, with a larger backbone.
89.3% macro accuracy · 0.896 AUROC · 0.285 calibrated NLL
Four benchmarks, equal-weight macro averages, November 2026. Visual Jev uses task SFT; API training data are unknown. TypeSafe Jev leads on ECE and TextVQA. NLL follows task-local temperature scaling. Results and protocol ↗
Full comparison and evaluation protocolTest images only (70% of each benchmark's images, split by a fixed hash of the image id); the same questions and option order for every system; each system gets its own temperature fitted on the same calibration images. TallyQA is the full test build with GQA-training duplicates removed. API probabilities are returned to two decimals and floored at 0.005; a failed request counts as a uniform guess. NLL and ECE after temperature scaling; AUROC of the top probability for correct vs. wrong answers. Macro is the equal-weight mean of the four benchmarks.
| System | GQA | SNLI-VE | TextVQA | TallyQA | Macro | NLL (TS) | ECE (TS) | AUROC |
|---|---|---|---|---|---|---|---|---|
| Visual Jev 8B | 0.919 | 0.818 | 0.979 | 0.855 | 0.893 | 0.285 | 0.021 | 0.896 |
| Visual Jev 4B | 0.916 | 0.808 | 0.975 | 0.841 | 0.885 | 0.309 | 0.025 | 0.890 |
| TypeSafe Jev | 0.885 | 0.713 | 0.982 | 0.851 | 0.858 | 0.361 | 0.013 | 0.870 |
| OpenAI Decisions | 0.834 | 0.732 | 0.955 | 0.818 | 0.835 | 0.440 | 0.020 | 0.833 |
We sent the same test questions, with the options in the same order, to two
hosted APIs that also return one probability per option: OpenAI's
GPT-6 Luna Decisions (openai/gpt-6-luna-decisions-20261006, through
OpenRouter) and TypeSafe Jev (jev-1.14.0, whose image input TypeSafe
marks as not yet production quality), queried in November 2026. Every system
gets its own temperature, fitted on the same calibration images.
Read it with three caveats:
†), the systems are close, and
TypeSafe Jev has the best TextVQA accuracy.Source: checked-in API comparison data.
CUDA or Apple Silicon MPS · Inference only · No training dataset required
git clone https://github.com/guanxuyu-sv/Visual-Jev.git
cd Visual-Jev
python3 -m venv .venv && source .venv/bin/activate
pip install torch torchvision
pip install -r code/requirements.txt
python code/examples/quickstart.py \
--model Qwen/Qwen3-VL-8B-Instruct \
--adapter guanxuyu/visual-jev-8b-answer-sft \
--image assets/figures/demo_bottle.jpg \
--request-file code/examples/bottle_questions.json --device auto
The runner downloads the backbone and this adapter on first use. --device auto selects CUDA first, then Apple MPS. For CUDA, install a compatible PyTorch build before the remaining dependencies; on Apple Silicon, use the regular macOS build shown above.
The request supplies named questions, shared state, and 2–16 candidate choices per question. Evaluation covers at most eight choices. The runner caches the shared input and returns a choice and probabilities for each question.
Explore the project’s interactive 4B reference examples ↗
Calibrating and acting on probabilitiesMacro over the same four benchmarks:
| Original 8B | This adapter | |
|---|---|---|
| NLL, raw | 0.634 | 0.298 |
| NLL, after temperature scaling | 0.376 | 0.285 |
| ECE (15 bins), after temperature scaling | 0.022 | 0.021 |
| AUROC, correct vs. wrong by top probability | 0.851 | 0.896 |
Post-training mainly improves ranking: confident answers are more often
right (AUROC, which does not depend on accuracy). It does not make the raw
probabilities calibrated; the fitted temperatures are all above 1, i.e. the
model is overconfident. A single temperature per task fixes that.
temperatures.json lists the temperature fitted for each seed and benchmark
on a separate calibration split; apply it as softmax(logits / T) over the
supplied options. For a new task, fit your own temperature on a few hundred
labelled questions from that task: in our measurements on the 4B adapter,
temperatures transferred from other tasks calibrated clearly worse.
Two further cautions before acting on the probabilities with a threshold:
incorrect_question or not enough evidence if you need one.Test images only (70% of each benchmark's images, split by a fixed hash of the
image id); mean ± standard deviation over three seeds. † marks a task family
no post-training saw.
| Benchmark | Original 8B | This adapter | 4B adapter | n test |
|---|---|---|---|---|
| GQA-Choice | 0.885 | 0.919 ± 0.001 | 0.916 | 5,447 |
| SNLI-VE | 0.695 | 0.818 ± 0.004 | 0.808 | 1,368 |
TextVQA-Choice † |
0.979 | 0.979 ± 0.003 | 0.975 | 1,742 |
TallyQA-Choice † |
0.853 | 0.855 ± 0.010 | 0.841 | 3,681 |
| Macro | 0.853 | 0.893 ± 0.003 | 0.885 |
The gain sits on the two task families represented in training; on the two held-out families the adapter matches the untrained backbone. The TallyQA column uses the full TallyQA test build (natural count distribution, with the images that are copies of GQA training images removed), distinct from the short count-balanced TallyQA subset in the original ablation evaluation. Compare this table with the full-build API comparison on the 4B card, not its original ablation table.
Load the adapter and reproduce the readoutimport torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
BASE = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER = "guanxuyu/visual-jev-8b-answer-sft" # seed 0 lives at the root
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,
device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(BASE)
# the other two seeds:
# PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")
The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position and candidate tokenisation determine the result; use the prompt construction in the code repository to reproduce the numbers above.
Training configuration and data| Base | Qwen/Qwen3-VL-8B-Instruct, vision tower frozen (exclude_modules: ".*visual.*") |
| Method | LoRA on the language tower, r=16, α=32, dropout 0.05 |
| Targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable | 43,646,976 parameters |
| Objective | full-vocabulary next-token cross-entropy at the readout position |
| Steps | 3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup, bf16 |
| Images | native resolution (see note) |
| Data | question records derived from GQA and SNLI-VE training splits; GQA Choice items presented with K in {2, 4} options |
| Seeds | 0 (root), 1 (seed1/), 2 (seed2/) |
Note on image resolution. The training configuration sets
max_pixels = 200,704 (196 visual tokens), but the Transformers image
processor used ignores that attribute, so training and every evaluation above
ran at the processor's default, which keeps native resolution (a 500×500 GQA
image becomes 256 visual tokens). The same correction applies to the 4B adapter.
No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. SNLI-VE is built on Flickr30k images, which are licensed for non-commercial research; using the backbone requires accepting its own licence.
@misc{yu2026visualjev,
title = {Visual Jev: Accurate and Efficient Decisions from Shared Visual Context},
author = {Guanxu Yu and Yuhang Yao},
year = {2026},
eprint = {2609.25845},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2609.25845}
}
This adapter is released under Apache-2.0. The backbone and source datasets retain their respective terms.
Reproduction guide · Experimental outputs · Brand asset sources
adapter_model.safetensors 175 MBseed1/adapter_model.safetensors 175 MBseed2/adapter_model.safetensors 175 MBQwen/Qwen3-VL-8B-Instruct