decider-12b-vision: typed decisions with calibrated probabilities on Gemma-4-12B, from text and images, one pass

decider-12b-vision answers typed decisions with a probability for every option. It reads each question in one forward pass and generates no text. The state may include images.

• Input: a state and one or more typed questions (Choice, Noul yes/no, Score), each with an explicit option list, in the System One wire format (POST /v1/systemone). Images are sent as Jev-style assets and referenced from the state.
• Output: a probability distribution over the options of every question.
• Weights: google/gemma-4-12B-it with a LoRA fine-tune (rank 32, attention and MLP) merged into the weights, bf16, about 23 GB. The vision tower is the base model's, unchanged.
• Code: decider.serve_vllm in decider-ai 1.10.0 for text and images; decider.serve and decider.infer.Decider for text.

Which 12B to use

decider-12b v2 stays the model we submit to JevBench: its yes/no answers score higher under JevBench's rule, and its config is set for that board. decider-12b-vision is better on the Decision Index task mix, reads images, and gives calibrated yes/no probabilities.

Results

All numbers measured by us; none were verified by the benchmark maintainers.

• The index sample is the 4,500-request sample of the 0.3 public suite; the full public suite was not run on this model. Through the released server the sample gives 56.63.
• Image questions: the eval splits of 11 sets from the_cauldron, 300 items each. Visual7W and VSR were never trained on. Through the released server with the image sent as an asset: Visual7W 0.880, VSR 0.800.
• Our own held-out sets against decider-12b v2: CLINC150 0.967 (0.903), iSarcasmEval 0.813 (0.707), VAST 0.793 (0.710), ACOS 0.987 / 0.853 (0.943 / 0.803), POP909-CL chords 0.690 (0.103), smart-home requests 0.998 (0.929).
• The temperatures were fitted on our own rows only; no Decision Index row, JevBench item or image evaluation item was used to fit or choose them.

Usage

python -m venv decider-vllm && . decider-vllm/bin/activate
pip install vllm==0.29.0 fastapi "uvicorn[standard]" jinja2 huggingface_hub pillow
pip install --no-deps "decider-ai>=1.10.0"
DECIDER_MODEL=Mapika/decider-12b-vision uvicorn decider.serve_vllm:app --host 127.0.0.1 --port 8000

A request with an image (the asset's key is the reference; the state names it):

import base64, httpx
img = base64.b64encode(open("receipt.png", "rb").read()).decode()
r = httpx.post("http://127.0.0.1:8000/v1/systemone", timeout=60, json={
    "state": {"receipt": "receipt-1", "policy": "Meals over 60 EUR need a manager's approval."},
    "assets": {"receipt-1": {"type": "image", "format": "png", "data_base64": img}},
    "questions": {"approval": {"type": "noul", "instructions": "Does the meal on `receipt` need approval?",
                               "criteria": {"true": "needs approval", "false": "no approval needed"}}}})
print(r.json()["answers"]["approval"])

Text-only use works the same without assets, and also through decider.serve or Decider("Mapika/decider-12b-vision") (text only; those start without image input and say so).

How a request is answered

1. First reading. Every question is rendered in the model's chat template with thinking off, and the log-probabilities of the option letters are read at the answer slot. A softmax at T1(n) = 2.078 − 0.036 ln n (n = number of options) gives the probabilities. Each image in the state is one placeholder, which vLLM expands to the image's soft tokens (at most 280).
2. Second reading, only when unsure (decider.serve_vllm): a question whose top probability is below 0.7 is read again with its options reversed; the two log-softmaxes are averaged and a softmax at T2(n) = 2.025 − 0.042 ln n gives the final probabilities.

Training

Two LoRA stages (rank 32, alpha 64) on google/gemma-4-12B-it, then merged:

1. 955 steps at learning rate 5e-5 (28.0M tokens, 2.5 hours on one B300) on 18,000 text rows from the decider-31b training mixture and 18,129 image rows.
2. 580 steps at 2e-5 (16.9M tokens) on the state-tracking mixture of decider-12b v2 (10,000 rows), plus 3,000 image rows and 4,000 text rows of stage 1 replayed.

• Image rows: train splits of seven the_cauldron subsets: AI2D, A-OKVQA, Hateful Memes, IconQA, RAVEN, ScienceQA, TQA. Yes/no questions are trained as Noul questions, the others as Choice questions.
• Text rows: the decider-31b mixture (train splits of CLINC150, iSarcasmEval, VAST, ACOS, POP909-CL, GSM8K and NLI4CT, decontaminated against every Decision Index 0.3 suite row; generated smart-home requests; generated state-tracking decisions and decision families; document questions written by Qwen3.6-27B; labelled rows from public datasets).
• Not used: Decision Index test rows, JevBench or ImageJevBench items.

Limitations

• One pass only: multi-step reasoning tasks are not solved by reading harder.
• Images: the gain is on the kinds of image questions trained; on new image question types it reads about as well as the stock model. Box answers and image references in question text are not supported. Not measured on ImageJevBench.
• Latency was not measured for this model.
• JevBench: use decider-12b v2 for that board (see above).

License

The weights are released under Apache 2.0, as the base model. Training sources and their terms:

• Image questions, from the train splits of the_cauldron subsets (the_cauldron licenses only its own prompts, CC BY 4.0; each source keeps its own terms):

  • AI2D (Kembhavi et al., 2016): CC BY-SA 4.0.
  • A-OKVQA (Schwenk et al., 2022): Apache 2.0 (repository); images from COCO.
  • IconQA (Lu et al., 2021): CC BY-NC-SA 4.0, non-commercial.
  • ScienceQA (Lu et al., 2022): CC BY-NC-SA 4.0, non-commercial research use.
  • TQA (Kembhavi et al., 2017): CC BY-NC 3.0, non-commercial.
  • RAVEN (Zhang et al., 2019): no data licence stated (the code is GPL-3.0).
  • Hateful Memes (Kiela et al., 2020): Facebook's Hateful Memes dataset licence. It permits use of the data to research and develop models designed to detect manipulated media, and states that models derived from it may be used for academic and commercial purposes. This is a research project; check the licence for your own use.

• CLINC150: CC BY 3.0 (Larson et al., 2019). Credit: "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction", clinc/oos-eval.

• iSarcasmEval: MIT (Abu Farha et al., 2022). The texts are tweets.

• POP909-CL: MIT; built on POP909 (Wang et al., 2020), MIT.

• VAST (Allaway and McKeown, 2020) and ACOS (Cai et al., 2021): no licence stated by the authors.

• GSM8K (Cobbe et al., 2021): MIT.

• NLI4CT (SemEval-2024 Task 2, Jullien et al.): no licence file in the task repository; released for the shared task.

• RACE: non-commercial research use. It is a small part of the replayed data (250 rows).

• ARC, SNLI, LogiQA 2.0 (copy dependent): CC BY-SA 4.0; BoolQ: CC BY-SA 3.0; BANKING77, MedQA: CC BY 4.0; MMLU, CommonsenseQA: MIT; MNLI: mixed per genre.