decider-2b-vision — option probabilities from an image in LiteRT

Give it one image and lettered questions, and it returns a probability for each option. For the published Pong frame game_pong_atari_up, asked "What should you do right now?" with the options move paddle up / move paddle down / stay, the upstream model in fp32 gives 0.99986 to "move paddle up" on the same 256×256 input. The fp16 and fp16-int8vocab files return the same value to five digits through the reference readout on a Mac CPU.

This is a third-party conversion of Mapika/decider-2b-vision at revision 863e290863655f1d6b69324d77d09ac972d21609. Mapika built it from the Qwen3.5-2B vision-language model with decider-2b v5 text weights (Qwen/Qwen3.5-2B-Base). The upstream model reads the letter logits at one answer slot per question and turns them into option probabilities; it does not generate answer text. Its training, results and limitations are on the upstream card. This repository changes the file format only.

The decoder has 24 layers: 18 gated-delta linear-attention layers and 6 full-attention layers. The vision encoder has 24 blocks and a 2×2 merger, so a 256×256 image becomes 64 tokens. The files were tested on LiteRT-LM 0.17.1 on a Mac (CPU and GPU); the int8 file was also tested on a Galaxy S26 with LiteRT-LM v0.16.0 (CPU and GPU). The reference readout needs ai-edge-litert 2.2.0, numpy, Pillow and tokenizers; it does not use torch or transformers.

Files

All three bundles hold the same tokenizer and prompt metadata: an identity template (no role markers), no start token, stop token 248044, one 256×256 image, a 4096-token cache, ExecutorMetadata for the 48 state buffers and an fp32 activation preference for the decoder. XNNPACK unpacks fp16 weights to fp32, so LiteRT-LM's CPU cache folder for one bundle holds 7,540,052,280 B (fp16 decoder), 6,016,360,768 B (fp16-int8vocab decoder) or 1,896,902,288 B (int8 decoder), plus 1,312,687,576 B for the vision encoder and adapter. With the GPU backend, the folder gets a decoder program cache that grew by 270,663,680 B at every engine creation in our runs (821,634,912 B after three), plus a 508,563,920 B weight cache for the int8 head of the fp16-int8vocab file. File checksums are in SHA256SUMS.

The int8 file quantizes activations on the fly on the CPU, and there its probabilities move: on the published image questions 51/53 answers match upstream, max |Δp| 0.429, p95 0.164, and the values also depend on how the prompt is split into prefill chunks (max |Δp| 0.233 with one padded chunk). On the Mac GPU (Metal, one padded chunk per answer slot) it computes the int8 weights in float: 52/53, max |Δp| 0.032, p95 0.011. Its probabilities on the phone were not measured, because the runtime returns text only; on the Galaxy S26 its first token was upstream's answer letter on 6/6 rows on both backends.

Use it

Option probabilities: the reference readout

python -m venv .venv
.venv/bin/pip install ai-edge-litert==2.2.0 numpy pillow tokenizers
.venv/bin/python -B reference/decider_litert.py --bundle decider-2b-vision_fp16-int8vocab.litertlm \
  --request reference/example_request.json

The example request is the Pong frame above. Output of that command on an Apple M4 Max:

{
 "model": "decider-2b-vision_fp16-int8vocab.litertlm",
 "backend": "cpu",
 "answers": [
  {
   "choice": "move paddle up",
   "confidence": 0.9998610273009231,
   "probs": {
    "move paddle up": 0.9998610273009231,
    "move paddle down": 1.4132675638724978e-05,
    "stay": 0.00012484002343821706
   }
  }
 ]
}

From Python:

import sys; sys.path.insert(0, 'reference')
from decider_litert import DeciderLiteRT

d = DeciderLiteRT('decider-2b-vision_fp16-int8vocab.litertlm')
d.decide('frame.png', 'You play Pong (Atari) and control the right paddle. ...',
         [{'question': 'What should you do right now?', 'options': ['move paddle up', 'move paddle down', 'stay']}])
# -> [{'choice': 'move paddle up', 'confidence': 0.99986..., 'probs': {...}}]

The first run copies the bundle's sections into .cache/readout/<bundle sha256>/ beside the reference/ folder (or --cache-dir) and writes the decoder's CPU weight cache there: 13,065,400,405 B in all for fp16 and 10,530,549,449 B for fp16-int8vocab. Pass image=None for a text-only request and several questions in one call; both are supported here and not through the runtime path below. backend='gpu' runs the decoder on the GPU with one padded prefill chunk per answer slot, for slots within the first 1024 tokens.

The answer letter: the LiteRT-LM runtime

pip install litert-lm==0.17.1 pillow tokenizers
python -B reference/runtime_example.py --bundle decider-2b-vision_fp16-int8vocab.litertlm \
  --image reference/fixtures/images/game_pong_atari_up_160x210.png \
  --context "You play Pong (Atari) and control the right paddle. Move the paddle so the ball hits it; the ball bounces off paddles and walls. Missing the ball loses a point. The image shows the current game screen." \
  --question "What should you do right now?" --options "move paddle up" "move paddle down" "stay"

runtime_example.py builds the prompt with the upstream prompt code, resizes the image to 256×256 with PIL, sends [image, text] as one user message and reads the first generated token with greedy decoding. This path covers one image first, one question per request, and the argmax only. The runtime returns text, not option probabilities, and its greedy token is the most likely token over the whole vocabulary, not only over the option letters. The two agree on every single-question published row. In one two-question row, the most likely token for a two-option question is "C".

Several questions about one image go into one conversation: the image with the first question, then each further question as a text-only message that starts with ) (it closes the previous letter) and carries the next Question: … Options: … Answer: ( block. Do not open a new conversation per request on the same engine: LiteRT-LM keeps this model's linear-attention state across conversations (LiteRT-LM #3165), and the answers drift. Measured on a Galaxy S26 through the Android library (litertlm-android 0.16.1, the int8 file, 2026-09-29): one conversation with five questions per bar chart gave the answer of a fresh engine on 18 of 20 questions on the GPU and 5 of 5 on the CPU; a new conversation per request gave it on 33 of 60 (GPU) and 34 of 60 (CPU) with pixel-identical inputs.

Examples

Published fixture images, one question each. The letter is the first token LiteRT-LM 0.17.1 generated on the Mac (fp16-int8vocab file, CPU and GPU backends, greedy); the probability is that option's, from the reference readout (fp16-int8vocab, CPU) and from upstream fp32 on the same 256×256 input.

What matches upstream

The files reproduce decider/vision.py prepare() + slot_logits() from the checkpoint under this contract:

1. One image, first in the prompt, resized to 256×256 with PIL bicubic. The vision encoder takes [0, 1] pixels and normalizes them itself; 64 image tokens follow <|vision_start|>.
2. The text is the upstream build() output with the option order kept, decoded and encoded again as upstream does. No BOS, no role markers.
3. Answer slots follow the upstream rule: a " (" token after ":", with "Answer" within the 5 tokens before it. The letter logits A–J at each slot are cut to the option count and softmaxed at T = 1.
4. Positions: the checkpoint uses 3-channel M-RoPE positions (image tokens on an 8×8 grid). LiteRT-LM passes one 1-D position, so the decoder derives the three channels from it inside its graph. On the 53 published image questions, plain 1-D positions flipped 2 answers and moved one probability by 0.198 in the upstream fp32 model; the files follow the M-RoPE values instead.
5. Text-only requests start at position 65, where the derived positions keep upstream's relative positions. Started at 0, the same 9 text-only questions moved by up to 0.0497 and one answer flipped in the fp32 LiteRT graph. The runtime starts text-only prompts at 0, so text-only requests go through the reference readout.

Agreement with upstream on the published rows

The fixtures hold 42 requests (37 with an image, 5 text-only; 62 answer slots) with the upstream fp32 letter logits and probabilities. Every image is a synthetic drawing made for this check: colour cards, shapes, text and simple Pong- and Breakout-style frames. check_fixtures.py rebuilds each request from the original image and text, asserts that the 256×256 pixels, token ids and answer slots equal upstream's, and compares the probabilities. |Δp| is the largest absolute probability difference over a slot's options. No upstream top-two gap is within the 1e-4 tie threshold.

The fp16-int8vocab CPU flip is v2_game_breakout_atari_fewbricks question 1, where upstream's top two options are 0.0073 apart. CPU rows are the included reference (ai-edge-litert 2.2.0 CompiledModel, 8 threads); its letter logits are bit-identical to the conversion's own graph readout on all 62 slots of all three files. GPU rows are ai-edge-litert 2.2.0 CompiledModel with GpuOptions(enforce_f32=True). Through LiteRT-LM 0.17.1 on the Mac, all three files on CPU and on GPU gave upstream's answer letter as the first token on 12/12 single-question image rows, with prefill counts equal to upstream's token counts and no Validation error lines.

The price of the fixed 256×256 input

Upstream picks a resolution per image; these files take 256×256. For 224×224, 256×240 and 256×256 originals, upstream's own processing produces the same pixels, so the probabilities are identical (13 published requests, 21 slots, |Δp| 0). Every other size gets fewer image tokens than upstream would use: 70 for the 160×210 frames and 144 to 475 for the larger synthetic images. On those published requests (24 requests, 32 slots), upstream fp32 on the 256×256 input matches upstream fp32 on the original image on 30/32 answers, with max |Δp| 0.373 and p95 0.318. On photographs and documents kept out of this repository, the largest move was 0.58. Both sides of these comparisons are upstream fp32, so the moves come from the input size, not from the conversion.

Performance

One decision per fresh process on an Apple M4 Max (Mac16,9, 128 GB, macOS 27.0) through LiteRT-LM 0.17.1 (PyPI, Python API): the published Pong frame (game_pong_atari_up, 256×256) with its one question, 150 prompt tokens including 64 image tokens, greedy, one output token. TTFT is the wall time from sending the message to the first token, measured in the process. Engine creation is the Engine(...) constructor. Peak RSS and peak footprint come from /usr/bin/time -l; RSS also counts the mapped model and cache files. CPU runs use 8 threads; GPU runs use Backend.GPU() (the WebGPU delegate on Metal) for the decoder and the vision encoder, after a rest of at least 120 s since the previous GPU run. Each value is the median of three runs, with the runs in parentheses. "none" is cache_dir=":nocache"; "yes, warm" runs after a first run that wrote the cache folder. Runs taken while another process used more than 120 % CPU were repeated or dropped and are not in the table. Every run gave the letter A, and the GPU runs printed no Validation error line. The int8 file was not timed on the Mac; its phone rows follow.

Median TTFT is 0.26–0.47 s on the GPU and 1.54–1.72 s on the CPU, apart from the fp16-int8vocab warm-folder CPU row explained below. GPU engine creation takes 27–30 s against 13–17 s on the CPU. Without a cache folder, the CPU process's peak footprint is 47.9 GB (fp16) and 41.3 GB (fp16-int8vocab); with a warm folder it is 6.6 GB and 1.5 GB. With a warm folder, CPU TTFT depends on whether the cache file is still in memory: 1.16–1.72 s when it was, 4.4–7.6 s when about 0.46–0.56 million page faults read it back from disk. A warm folder did not shorten GPU engine creation. The fp16-int8vocab GPU warm row has two runs; its third run overlapped another process and was dropped.

The reference readout on the same machine (ai-edge-litert 2.2.0 CompiledModel CPU, 8 threads, its section folder and weight cache already on disk), one process per run. The first decision in a process includes the first use of each graph; the second decision in the same process is the steady cost.

Galaxy S26 (SM-S942Q, MemTotal 11,389,756 kB), LiteRT-LM v0.16.0 litert_lm_advanced_main, a different runtime version from the Mac rows. One cold run per row (n = 1): the published row game_pong_atari_level (150 prompt tokens, one question), greedy, up to 3 output tokens, runtime default cache folder, thermal status 0 and no frequency cap before the run. Engine creation is the runtime's Init Total; the conversation set-up after it is in parentheses. Peak memory is the process VmHWM.

On the GPU all 7 decoder signatures were fully delegated to OpenCL (one partition each); the vision encoder ran on the GPU and the adapter on the CPU (the cache files each wrote). VmHWM does not include GPU memory: during the int8 GPU run the phone's MemAvailable fell to 0.77 GB, so the int8 GPU run is close to the limit of a phone of this RAM class. The other five int8 CPU rows peaked at 4.82–4.85 GB VmHWM. Every int8 row on the S26 printed no Validation error line and processed the same number of prefill tokens as upstream.

Limitations

• The runtime path gives the argmax, with one image first; several questions about that image go as turns of one conversation (above), not as one conversation per question. The option probabilities cannot be read through LiteRT-LM: its text-scoring call returns scores that disagree with its own greedy decode on 0.17.1 (LiteRT-LM #3561). Use the reference readout for probabilities and for text-only requests.
• The runtime's own image resize was not exercised: every runtime check handed it a 256×256 PNG resized with PIL. The examples here do the same.
• One image per request. English only. The text behaviour is the upstream model's (decider-2b v5 text weights); see the upstream limitations.
• Rows longer than 4096 tokens do not fit the cache. Every published row is at most 225 tokens; prompts above 2048 tokens were not exercised.
• Phones: only the int8 file was tested on a phone (Galaxy S26, one cold run per row). The fp16-int8vocab file did not fit its GPU, and the fp16 file was not tried. Probabilities on the phone were not measured.
• 8 GB-class phones and Mali. The _int8 file's GPU engine is verified on the Galaxy S26 (Adreno) only. During that GPU run, free memory on the phone (MemTotal 11,389,756 KiB) fell to 0.77 GB. So the GPU engine is not expected to fit an 8 GB-class phone — an expectation from that run, not a measurement on such a phone. No Mali phone has run any of the three files. Use the CPU backend on 8 GB phones for the _int8 file; on the CPU its probabilities move, as the CPU figures above show.
• GPU engine creation does not get faster after the first. On a Galaxy S26 (OpenCL) the program cache did not shorten GPU engine creation of the _int8 file: in six processes run one after another with LiteRT-LM v0.16.0 and the runtime's cache kept beside the file (the Performance row above is the first of them), each process after the first loaded 3 of the 8 GPU programs from the cache (prefill_1, decode and the vision encoder), failed to load the other 5 (the multi-token prefill signatures, DATA_LOSS) and compiled those again, and its engine took 71–84 s to create against 64 s for the first. On the Mac GPU (LiteRT-LM 0.17.1, WebGPU) a warm cache folder did not shorten engine creation of the _fp16 and _fp16-int8vocab files either (Performance above; the _int8 file was not timed there), and their decoder's program cache grew by about 270 MB per creation (550,971,232 B after two creations, 821,634,912 B after three). These files have not run on an iPhone, so the iOS GPU is not verified; the base model's own file fails the same way on an iPhone 17 Pro (Metal, LiteRT-LM v0.18.0; LiteRT-LM #4178, Qwen3.5-2B).
• Tool calls. All three files carry an identity prompt template that renders the request text as is, so no tool definitions are rendered into the prompt. On litert-lm 0.18.0 the runtime also parses no Qwen3.5 tool call: such a call comes back as plain text and tool_calls stays empty (my open report).

Other ports

As of 2026-09-29, the Hugging Face search for decider-2b-vision lists two GGUF ports, mradermacher/decider-2b-vision-GGUF and mindchain/decider-2b-vision-GGUF, and no other LiteRT port.

Known runtime issues

Mac GPU, long outputs: not verified on these files. On a Qwen3.5-4B build (int8 and Mixed INT4) on an Apple M4 Max, the default GPU path, where the sampler also writes the next decode step's inputs (the log says Sampler will handle decode input tensors.), gave a different greedy reply on each run and emitted tokens the model itself scores at a log probability of −16 to −23 (litert_lm_advanced_main built from LiteRT-LM main at dab2e708); with litert-lm 0.19.0 from PyPI, which always takes this path, 16 of 22 long code replies failed node --check. With --sampler_handles_input=false, or --sampler_backend=cpu, on that C++ CLI the reply was the same on every run and 0 of 7 failed. The Python, Swift, Kotlin and C APIs of 0.19.0 have no setting for this.

License and changes

Apache-2.0, as declared by the upstream checkpoint; the vendored prompt code is Apache-2.0 too (LICENSE). Source revision: 863e290863655f1d6b69324d77d09ac972d21609.

Changes: converted the text decoder, the vision encoder and the merger to LiteRT flatbuffers; the decoder derives the upstream M-RoPE positions from a 1-D position inside its graph; fixed the image input at 256×256; cast the weights as listed under Files; replaced the chat template with an identity template; repackaged the tokenizer unchanged; added ExecutorMetadata and an fp32 activation preference. The multi-token-prediction head is not included. This community conversion is not an official Mapika or Qwen release.

decider_vendored/prompt.py is the checkpoint's own decider/prompt.py, unchanged (SHA-256 c2fadbe0e4703a381011007f52103ab29142a55a441302d62c1dfb2056acce16). decider_vendored/infer.py keeps only its Q and Example dataclasses. The LICENSE is from github.com/Mapika/decider at commit 23579f7a7e8f10e1045be492af3c1c05a005d67c.

Reproduction (conversion scripts, gates, fixtures): hf-to-litertlm decider2bv_work/.