Shisa DE-2

Shisa DE-2 (Decision Engine 2) is a general-purpose decision model. It takes one state and a list of named questions and returns one typed answer per question: yes/no, or one of up to 256 options, each with a probability. No answer text is generated. Each answer is read from the next-token distribution at the answer boundary, restricted to the option codes.

DE-2 is based on Gemma 4 26B A4B with a rank-16 LoRA on the language model's attention projections merged into the weights. It was trained with softmax cross-entropy over the restricted option distribution, which is the same readout used at inference. The vision encoder is present in the weights and was not specifically trained, but which has been evaluated to function extremely well for decision tasks.

What changed from DE-1

DE-2 supercedes our prior Shisa DE-1 model with some important upgrades:

The Shisa DE models are designed to run on existing inference engines (vLLM, SGLang, llama.cpp, etc) without requiring any modifications. You can manually interact with the models using the readout contract and inference policy described below, however we also have a Shisa DE SDK that makes interaction easier.

Model details

Serving

Readout contract

Send one request per question. Render the state, the question and the option list with the chat template, then read logprobs at the answer position with max_tokens=1. Restrict the logits to the valid option codes and normalize. Take the argmax for a choice answer, or the probability of the Yes row for a noul answer. The model's unrestricted next-token distribution is not part of the contract.

The scaffold is a system instruction and a JSON payload in the user message:

<bos><|turn>system
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.<turn|>
<|turn>user
{"evidence": <state>, "criterion": <question>, "options": [{"letter": "A", "description": ...}, {"letter": "B", "description": ...}]}<turn|>
<|turn>model

Options are labelled A to Z, then AA, AB and so on up to 256. Every code is a single token. A noul question uses the option pair Yes/No.

Launch

DE-2 serves on mainline vLLM with no special options:

vllm serve shisa-ai/shisa-de-2

Then use the Shisa DE SDK, which applies the readout contract, the adaptive policy and the calibration below:

from shisa_de import DecisionModel

de = DecisionModel.from_endpoint("http://127.0.0.1:8000/v1", model="shisa-ai/shisa-de-2", api_key="")
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})

Requirements

• vLLM attention backend. Pass attention_config={"backend": "TRITON_ATTN"} on Hopper GPUs (H100, H200, H20) vLLM 0.30.0 otherwise runs this base model's 512-wide attention heads through a FlashAttention 4 path that lowers accuracy: 48.59 against 52.58 on a Decision Index run for the same weights. Triton and FlashInfer tested on SM90, SM120 did not have any accuracy issues.
• String content for text questions sent through the chat endpoint. The SDK reads text questions through /v1/completions and needs nothing here. If you send text questions through /v1/chat/completions yourself, serve with --chat-template-content-format string: the unmodified Gemma 4 chat template writes a space after the system text when the system message arrives as content parts and none when it arrives as a string, and DE-2 was trained and evaluated on the string rendering. Leave the option unset if you send images; with it set, vLLM 0.30.0 rejects image requests.
• Sampling defaults. Read probabilities with temperature 0 or 1 and without top_k or top_p truncation.

Adaptive inference policy

Based on empirical testing, we recommend the following adaptive inference policity (applied by the SDK automatically):

1. Write the user turn twice, separated by \n\nRead the same input again before answering:\n, and read the option codes.
2. If the top probability is below 0.7 and the question has 26 options or fewer, ask again with thinking on, let the model reason for up to 1,024 tokens, close the thought and read the codes after it. Otherwise keep the answer from step 1.

The SDK's two steps are worth 4.3 points on the Decision Index. The same weights score:

Reading twice helps every area. Thinking adds most where the task needs working out: Knowledge & Reasoning gains 3.8 points from it, and skill on GSM8K goes from 70.2 to 81.4. The policy thinks on 4% of questions.

There is of course slight latency penalty so if fast or predictable response is required, one or both of these policies can be disabled:

Response times are measured one request at a time on one H20-3e with a cold prefix cache.

A chat template with an opt-in repeat_input=true argument is provided for callers who serve through a plain chat endpoint. It covers step 1 only, is off by default, repeats the last user turn only, and must not be combined with a client that already repeats the input.

Calibration

Raw DE-2 probabilities are overconfident, and by a different amount on each read. A post-hoc temperature is fitted per read and per answer kind by NLL grid search, and the SDK applies the one for the read each answer came from:

Fitted on 9,931 questions from 53 of our evaluation suites (84% English, 16% Japanese), each suite weighted equally, served by vLLM 0.30.0 on one RTX PRO 6000. The single-read and read-twice temperatures were fitted on half of the question groups and measured on the other half (1,895 noul and 3,140 choice questions); the thinking temperature is one value fitted on 475 thoughts and measured by cross-validation. Over the adaptive policy's whole output, ECE moves from 0.091 to 0.021. Accuracy is unchanged, because tempering does not move the argmax.

Probabilities read after thinking remain the least reliable: those answers are right 52% of the time at 92% raw confidence. For questions scored on the probability itself, such as forecasts, use the single read: reading twice and thinking both shift forecasting answers toward "yes", which a temperature does not correct. Refit on your own rows before using these probabilities to drive a threshold.

Training

The 28 training families, by group:

• Distilled and procedural decision data in English and Japanese.
• Japanese judgement families: leakage of internal information, nuance, policy, register, and indirect refusals in context.
• Fraud and safety: antiscam messages with link and sender cues, prompt injection and hidden text.
• Agent workflow: compaction, guardrails, review triage and progress checks.
• Wide-option and ordinal tasks: large-option financial intents, relation and sentiment, product, emotion and sentiment ratings.
• Exactness and logic: sibling criteria where one detail decides the answer, compound conditions, counting and arithmetic probes.
• A second classifier-benchmark suite of 49 short tasks.

Evaluation

Unless a table says otherwise, each system is shown at its best measured reading: DE-2 through the SDK's reading policy, DE-1 with the SDK's scaffold, Jev as the hosted jev-1.13.0 API, and OpenAI Decisions (gpt-6-luna) via the latest OpenAI SDK. All DE-2 and DE-1 figures are our own local runs. A dash means the system was not run.

Summary

• Decision Index 0.3. Full public suite, 140,178 scored requests, official scorer; every request is answered. These are not board entries. The board ranks by a Full score that adds private tests only its maintainers run; Jev's public-index figure is from the board on 2026-10-06. Other public-index figures there: Clef 61.71, Rune 26B-A4B v3 58.28, Decider chat on Gemma-4-31B 57.79.
• Plan-RewardBench. 1,171 pairs of tool-agent trajectories, both presentation orders averaged, following the public JEV-27B-DEMO script. The stock base scores 76.1%, and the JEV-27B-VL card reports 73.2%.
• Image benchmarks are in "Vision" below. Jev 1.13.0 has no vision encoder, so the comparisons there are with vision-language decision models.

JevBench

The 231 public JevBench items, by tier:

"Medium" is JevBench's standard tier. DE-2 and DE-1 use the Shisa DE SDK, with the best policy results chosen. Jev and OpenAI Decisions (gpt-6-luna) are their hosted APIs on the same items, one request per item. The easy and medium tiers are saturated for all four models.

Harness suites

67 internal suites were run against DE-2, DE-1 and hosted Jev and OpenAI Decisions on the same questions, read once. Primary metric per suite; a selection:

On the 64 suites all four systems answered, DE-2 has the top result on 38, Jev on 29, DE-1 on 18 and OpenAI Decisions on 8, counting a tie for each system that shares it. Several of the suites where DE-2 leads share a task family with its training data. DE-1 keeps an edge on the original four use cases it was built for.

Response time per case on those 64 suites (1.4 questions per case on average):

DE-2 and DE-1 were served locally by vLLM on one H20-3e, read once, with one client sending requests in sequence. Jev and OpenAI Decisions are hosted APIs measured from our client, network round trip included; the OpenAI runs had up to 16 requests in flight.

Vision

Like DE-1, DE-2's vision encoder is the base model's and was not trained, but the decision readout carries over to images. Jev 1.13.0 takes no image input. OpenAI Decisions (gpt-6-luna) was run by us on the same items; the Qwen3-VL-4B and JEV-27B-VL figures are their authors' published numbers.

Image classification, glance suites (one forward pass per question):

Multimodal judging, following the public JEV-27B-DEMO protocols with both presentation orders averaged:

The stock base model scores 80.0%, 60.0% and 0.907 on the three judging benchmarks, so on images DE-2 is the base model's vision with DE-2's readout; training neither helped nor hurt. On Multimodal RewardBench 2, JEV-27B-VL is ahead on text-to-image (69.2% against 64.1%) and interleaved judging (67.8% against 57.8%), and DE-2 on image editing (65.0% against 57.8%).

Limitations

• Overconfident raw probabilities. The SDK's calibration corrects most of this for the single and read-twice answers. Probabilities read after thinking stay poorly calibrated (ECE 0.18 after tempering), and calibration was not fitted for image questions.
• Forecasting questions. On ForecastBench (retired from the index in edition 0.3) skill was 5.5 read once and 0 under the full policy, because the model says "yes" too often. Use the single read with its calibration when the probability is what you need.
• Rare-positive detection. When very few candidates are true, as in aspect extraction with dozens of pairs per input, the model raises many false alarms: precision is about 10% on ACOS.
• Order sensitivity on multi-image comparisons. Swapping two candidate responses changes the verdict on 45% of Multimodal RewardBench 2 pairs. Ask both orders and average.
• Agent workflow and routing. DE-2 is 3 points below the stock base and DE-1 on Plan-RewardBench, behind Jev on the compaction, guardrails and progress suites, and tends to route simple tasks to a larger tier than needed.
• Knowledge limits. Scores on expert-knowledge and game benchmarks (HLE, GPQA Diamond, ChessBench) do not improve with either reading.
• Thinking is limited to 26 options. Wider questions use the read-twice answer.
• Prompts longer than 65,536 tokens were not evaluated, however runs with the max-context length (256K) vs restricted (32K) showed no regressions.