Shisa DE-2.1

Shisa DE-2.1 (Decision Engine 2.1) is a general-purpose decision model. It takes one state and a list of named questions and returns one typed answer per question: yes/no, or one of up to 256 options, each with a probability. No answer text is generated. Each answer is read from the next-token distribution at the answer boundary, restricted to the option codes.

DE-2.1 is based on Gemma 4 26B A4B. Its weight update is the average of eleven rank-16 LoRA runs on the language model's attention projections, trained with different seeds and merged into the weights. Each run was trained with softmax cross-entropy over the restricted option distribution, which is the same readout used at inference. The vision encoder is the base model's and was not trained.

DE-2.1 is a drop-in replacement for Shisa DE-2: the prompt layout, the readout contract and the inference policy are unchanged, and only the weights differ.

What changed from DE-2

DE-2.1 started as an experiment testing out some of RSI-Jev's training techniques. Of those, only the model-souping had a measurable positive impact. While reviewing the results, we also made some improvements in data draw and ordering, as well as some targeted data improvements: rebalancing select data labels, adding tool-availability, none-option and evidence-mention supervision, and strengthening hidden-text training with scanner-labeled untrusted-data examples.

DE-2's published Decision Index of 58.96 is from another run of the same weights; two runs differ by about 0.15.

The Shisa DE models are designed to run on existing inference engines (vLLM, SGLang, llama.cpp, etc.) without requiring any modifications. You can interact with the models directly using the readout contract and inference policy described below; the Shisa DE SDK makes this easier.

Model details

Serving

Readout contract

Send one request per question. Render the state, the question and the option list with the chat template, then read logprobs at the answer position with max_tokens=1. Restrict the logits to the valid option codes and normalize. Take the argmax for a choice answer, or the probability of the Yes row for a noul answer. The model's unrestricted next-token distribution is not part of the contract.

The scaffold is a system instruction and a JSON payload in the user message:

<bos><|turn>system
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.<turn|>
<|turn>user
{"evidence": <state>, "criterion": <question>, "options": [{"letter": "A", "description": ...}, {"letter": "B", "description": ...}]}<turn|>
<|turn>model

Options are labelled A to Z, then AA, AB and so on up to 256. Every code is a single token. A noul question uses the option pair Yes/No.

Launch

DE-2.1 serves on mainline vLLM with no special options:

vllm serve shisa-ai/shisa-de-2.1

Then use the Shisa DE SDK, which applies the readout contract, the adaptive policy and the calibration below:

from shisa_de import DecisionModel

de = DecisionModel.from_endpoint("http://127.0.0.1:8000/v1", model="shisa-ai/shisa-de-2.1", api_key="")
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})

Use shisa-de 0.6.0 or later, which bundles DE-2.1's calibration; earlier versions read DE-2.1 correctly but return raw probabilities.

Requirements

• vLLM attention backend. Pass attention_config={"backend": "TRITON_ATTN"} on Hopper GPUs (H100, H200, H20): vLLM 0.30.0 otherwise runs this base model's 512-wide attention heads through a FlashAttention 4 path that lowers accuracy (measured on DE-2: 48.59 against 52.58 on a Decision Index run for the same weights). Triton and FlashInfer tested on SM90, SM120 did not have any accuracy issues.
• String content for text questions sent through the chat endpoint. The SDK reads text questions through /v1/completions and needs nothing here. If you send text questions through /v1/chat/completions yourself, serve with --chat-template-content-format string: the unmodified Gemma 4 chat template writes a space after the system text when the system message arrives as content parts and none when it arrives as a string, and DE-2.1 was trained and evaluated on the string rendering. Leave the option unset if you send images; with it set, vLLM 0.30.0 rejects image requests.
• Sampling defaults. Read probabilities with temperature 0 or 1 and without top_k or top_p truncation.

Adaptive inference policy

Based on empirical testing, we recommend the following adaptive inference policy (applied by the SDK automatically):

1. Write the user turn twice, separated by \n\nRead the same input again before answering:\n, and read the option codes.
2. If the top probability is below 0.7 and the question has 26 options or fewer, ask again with thinking on, let the model reason for up to 1,024 tokens, close the thought and read the codes after it. Otherwise keep the answer from step 1.

The same weights score:

The SDK's two steps are worth 4.4 points on the Decision Index.

There is of course a slight latency penalty, so if a fast or predictable response is required, one or both of these steps can be disabled:

One RTX PRO 6000 (TP=1):

Two RTX PRO 6000s, tensor-parallel (TP=2):

Response times are measured one request at a time with vLLM default settings, on one RTX PRO 6000 (TP=1) or tensor-parallel over two (TP=2), on 700 requests of the Decision Index board's sample mix after 10 warmup requests. A second GPU mostly shortens thinking. A read costs the same as DE-2's; under the full policy DE-2.1's average is higher (DE-2: 886 ms at TP=1, 610 ms at TP=2) because it thinks on more requests, 111 of the 700 against DE-2's 80.

A chat template with an opt-in repeat_input=true argument is provided for callers who serve through a plain chat endpoint. It covers step 1 only, is off by default, repeats the last user turn only, and must not be combined with a client that already repeats the input.

Calibration

Raw DE-2.1 probabilities are overconfident, less so than DE-2's, and by a different amount on each read. A post-hoc temperature is fitted per read and per answer kind by NLL grid search, and the SDK applies the one for the read each answer came from:

Fitted on all 19,597 questions of 59 of our evaluation suites, each suite weighted equally, served by vLLM 0.30.0 on one RTX PRO 6000. The single-read and read-twice temperatures were fitted on half of the question groups and measured on the other half (3,612 noul and 6,240 choice questions); the thinking temperature is one value fitted on 1,298 thoughts and measured by 5-fold cross-validation. Accuracy is unchanged, because tempering does not move the argmax.

DE-2's temperatures (2.69 / 1.81 single, 2.76 / 1.89 twice, 6.8 after thinking) do not fit DE-2.1: on the same held-out half they leave ECE at 0.034 / 0.045 single and 0.048 / 0.042 twice, and make read-twice yes/no answers worse than raw (0.040). After thinking, DE-2's 6.8 gives 0.064 on these rows against 0.086 for the cross-validated refit. Probabilities read after thinking remain the least reliable: those answers are right 62% of the time at 94% raw confidence. Refit on your own rows before using these probabilities to drive a threshold.

Training

The 31 training families, by group:

• Distilled and procedural decision data in English and Japanese.
• Japanese judgement families: leakage of internal information, nuance, policy, register, and indirect refusals in context.
• Fraud and safety: antiscam messages with link and sender cues, prompt injection and hidden text.
• Agent workflow: compaction, guardrails, review triage and progress checks.
• Tool use: whether a listed tool fits the request, and when to decline, ask or call (new in DE-2.1).
• Wide-option and ordinal tasks: large-option financial intents, relation and sentiment, product, emotion and sentiment ratings.
• Exactness and logic: sibling criteria where one detail decides the answer, compound conditions, counting and arithmetic probes, none-of-the-above options and evidence mentions (the last two new in DE-2.1).
• A second classifier-benchmark suite of 49 short tasks.

Evaluation

Unless a table says otherwise, each system is shown at its best measured reading: DE-2.1 and DE-2 through the SDK's reading policy, Jev as the hosted jev-1.13.0 API, and OpenAI Decisions (gpt-6-luna) via the latest OpenAI SDK. All DE-2.1 and DE-2 figures are our own local runs; the Jev and OpenAI Decisions figures are the runs reported on the DE-2 card, on the same items. A dash means the system was not run.

Summary

• Decision Index 0.3. Full public suite, 140,178 scored requests, official scorer; every request is answered. These are not board entries. The board ranks by a Full score that adds private tests only its maintainers run; Jev's public-index figure is from the board on 2026-10-06. Other public-index figures there: Clef 61.71, Rune 26B-A4B v3 58.28, Decider chat on Gemma-4-31B 57.79.
• Plan-RewardBench. 1,171 pairs of tool-agent trajectories, both presentation orders averaged, following the public JEV-27B-DEMO script.

Decision Index 0.3 by area

JevBench

The 231 public JevBench items, by tier:

"Medium" is JevBench's standard tier. DE-2.1 and DE-2 are read through the full policy with DE-2's scaffold; Jev and OpenAI Decisions are their hosted APIs on the same items, one request per item. The easy and medium tiers are saturated. On the hard tier, by reading:

Harness suites

66 internal suites, run against both models on the same questions under the three reads, questions of a case asked together through the SDK. Mean of each suite's primary metric:

Single read through the harness, the same backend and questions as the DE-2 card's runs; primary metric per suite, a selection:

On the 63 suites all four systems answered, DE-2.1 has the top result on 35, Jev on 28, DE-2 on 19 and OpenAI Decisions on 7, counting a tie for each system that shares it.

Vision

DE-2.1's vision encoder is the base model's and was not trained, as in DE-2, so on images it is the base model's vision with DE-2.1's readout.

Image classification, glance suites (one forward pass per question):

DE-2.1 and DE-2 were run on the same machine and glance checkout (200 / 200 / 100 / 50 test items); the differences between them are a few items each. Jev 1.13.0 takes no image input; the Qwen3-VL-4B figures are its authors' published numbers.

Multimodal judging, following the public JEV-27B-DEMO protocols with both presentation orders averaged:

OpenAI Decisions (gpt-6-luna) was run by us on the same items; the JEV-27B-VL figures are its authors' published numbers.

Limitations

• Not every eval improved over DE-2. Notably When2Call tool-call (88.3% vs 91.0% read once) and hidden-text v2 read twice (87.1% vs 91.4%).
• Overconfident raw probabilities. The SDK's calibration (shisa-de 0.6.0 and later) corrects most of this for the single and read-twice answers; DE-2's temperatures over-correct DE-2.1. Probabilities read after thinking stay the least reliable, and calibration was not fitted for image questions.
• Forecasting questions. On ForecastBench (retired from the index in edition 0.3) Brier skill is 24.6 read once, 25.3 read twice and 20.4 under the full policy (DE-2: 8.0, 0 and 0 on the same runner); thinking lowers it. Use a read without thinking when the probability is what you need.
• Rare-positive detection. When very few candidates are true, as in aspect extraction with dozens of pairs per input, the model raises many false alarms: precision is about 10% on ACOS (9.6% read once, 12.2% under the policy).
• Order sensitivity on multi-image comparisons. Swapping two candidate responses changes the verdict on 40% of Multimodal RewardBench 2 pairs. Ask both orders and average.
• Knowledge limits. Under the policy HLE (10.8 against DE-2's 11.8) and ChessBench (21.2 against 20.6) barely move; GPQA Diamond rises from 44.9 to 51.0.
• Thinking is limited to 26 options. Wider questions use the read-twice answer.
• Prompts longer than 65,536 tokens were not evaluated separately.