decider-2b: typed decisions with calibrated probabilities in one forward pass

A language model that does not generate text. It reads a state and one or more typed questions, each with an explicit option list, and returns a probability distribution over the options for every question from one forward pass. There is no decoding, no parsing and no output outside the options you defined. It is called from software, not chatted with. It is an open reproduction of the "System One" model class (TypeSafe AI's Jev).

Base model: Qwen/Qwen3.5-2B-Base (1.9B parameters). The supervised stages (v1 to v8) fine-tune it with cross-entropy, a proper scoring rule, on a mixture of about 95 public decision datasets, agent trajectories, web element choice, game states and teacher-written custom questions, in two prompt layouts and with isolated Score levels. v10 continued the v8 weights for 384 steps of calibration-aware reinforcement learning whose only rewards are outcomes (live browser task checkers and the exact probability laws of games), with a hard KL limit to the v8 weights on replayed training rows. This repository holds v11: v10 plus a LoRA of rank 64, trained for 2 epochs on 42,749 rows of harder decisions and replay, with the replay rows trained toward v10's own answer distribution, merged into the weights, and one temperature per answer type. v10 stays available under the Hub tag v10 and v8 under v8. Code, data registry, training scripts and the recipe are at https://github.com/Mapika/decider; decider/ in this repository is the inference subset of that package. The other sizes and the vision variant are listed under The decider family.

v11 is better than v10 on hard decisions and worse on some everyday and game rows. On the same rows: held-out generated decision families 0.429 against 0.324, held-out document questions 0.753 against 0.646, JevBench public hard tier 0.577 against 0.459, and better calibrated on both hard sets than v10 (0.156 against 0.226 and 0.175 against 0.307), though still overconfident there. Losses: human-labelled public sets −2.2 points, a knowledge guard set −1.6, greedy bag-draw play −10.9, sampled slippery-grid play −4.3, sampled browser play −2.8 (interval includes zero), TypeSafe −4.9 (interval includes zero). Details under Changes from v10.

Contents: Changes from v10 · The decider family · Usage · How it works · Field types · Training · Evaluation · Speed · Limitations · Changelog · Reproduction

Changes from v10

v11 is v10 plus one supervised LoRA stage (described under Training) and a per-type temperature map:

• Harder decisions. The same 22,649 labelled rows as the stage 2 of decider-4b v2.1: 8,000 rows from ten generated decision families whose answers the generating code computes, 11,356 questions over business documents written by Qwen3.6-27B and kept when two further independent answers agreed, and 3,293 human-labelled rows.
• Replay trained toward v10's own distribution. 20,100 replay rows from the public decision mixture, trained toward v10's answer distribution (loss KL(p_v10 ‖ p_model) over the options) instead of their labels, so that everyday answers stay close to v10's. On 1,000 held-out replay rows the mean KL at temperature 1 is 0.047 nats and 94.9% of the argmax answers equal v10's. The fitted temperature is 1.145 (v10 1.30).
• One temperature per answer type. decider_config.json has temperature 1.145 and temperature_by_type {"choice": 1.164, "noul": 1.624, "score": 1.124}, fitted by NLL with decider.calibrate (decider-ai 1.4.0). decider-ai 1.4.0 and later use the map. decider-ai 1.3.0 and earlier ignore the map and serve every answer at 1.145; a temperature does not change which option is most probable, so the answers are the same either way (except exact ties between the level rows of an isolated Score answer: 1 row of 5,000 on the held-out generated families and 1 of 944 on the teacher validation rows changed); the probabilities differ, mostly on yes/no answers.
• The model name in answers is decider-2b-v11 (v10 reported decider-v10).
• decider_config.json no longer marks the model as trained for the schema-first layout (schema_first_trained: false; v10: true), because the LoRA stage used only the state-first layout and no schema-first temperature was fitted. With DECIDER_SCHEMA_CACHE=1 the server therefore does not turn the schema cache on for v11. Decider.schema() still runs; its accuracy was not measured on v11.

All rows below are on identical inputs and seeds: v11 through decider-ai 1.4.0 with its map, v10 through decider-ai 1.3.0 at its stored temperature 1.30, in the same session on 2026-09-24 (so the v10 numbers can differ slightly from the ones first published for v10 further down). Intervals are 95% paired bootstrap intervals (rows for the fixtures, boards for the games, task-seed pairs for the browser). The regression set and our own sets were read from stored temperature-1 logits at each model's served temperatures. v10's JevBench file was read earlier through decider-ai 1.2.1; accuracy does not depend on the temperature.

Gains. On the two held-out hard sets, whose families and business domains are not in the training data, v11 is 10.5 and 10.7 points above v10, and on JevBench's public hard tier 13 items (11.7 points) above it. OpenJev +1.4 and Mind2Web +1.3 points (intervals exclude zero), command-risk probe +4 items. Calibration on hard items is better than v10's: 0.156 against 0.226 on the held-out generated families and 0.175 against 0.307 on the JevBench hard tier.

Losses, stated plainly.

• Human-labelled public sets (the validation halves of MMLU, ARC, CommonsenseQA, BoolQ, MNLI, SNLI, Banking77 and others): 0.781 against 0.803, −2.2 points. A knowledge guard set (MMLU, ARC and similar): 0.715 against 0.732, −1.6 points.
• Greedy bag-draw play: 46.9% against 57.8% wins, −10.9 points (interval −20.3 to −3.1). Sampled slippery-grid play: 14.8% against 19.1%, −4.3 points (interval −7.8 to −1.2).
• Sampled browser play: 90.3% against 93.2%, −2.8 points (interval −7.4 to +1.1, includes zero); on the 16 rewarded tasks −3.9 (interval −9.4 to +0.8). Greedy browser play and the six held-out tasks are level.
• TypeSafe workflow decisions −4.9 points (interval −10.8 to +1.0), 847 validation rows −1.4 (−3.0 to +0.1), regression set −0.4 in-task and −0.3 held-out; needs-live-data probe one item lower.
• Still overconfident on hard items: calibration error 0.156 on the held-out generated families, where our release limit is 0.08.
• The issue #9 form cases are 0 of 4, as for v10.

How v11 was chosen, and why it is released although it did not pass. The run had a pre-registered release rule for the 2B. A checkpoint was eligible only if it lost at most 1 point on the human-labelled sets against v10; every checkpoint lost more (v11: −2.2), so no checkpoint was eligible. The rule's fallback candidate is v11 (two arms tied within 0.002 on the selection score; the tie went to the higher regression in-task accuracy). It passes the numeric items (regression accuracy at most 1.0 point in-task and 0.5 held-out under v10, held-out generated families at least 5 points and held-out document questions at least 3 points above v10, regression ECE at most 0.01 above v10's, sampled browser, zero-shot and bag-draw play not shown to be below v10's: upper end of the 95% interval at least 0) and fails the calibration item: calibration error on the held-out generated families at most 0.08. v11 is at 0.171 at its global temperature and 0.156 with the map; v10 is at 0.226 and would fail the same item. A second pre-registered rule for the temperature map had the same 0.08 limit, so the map did not pass either. v11 is released with the map on a decision made after reading the full comparison above. The JevBench public items were read once for v11 at its global temperature and once with the map, after the rule decisions; they were not used for training, selection or the temperatures.

Which version to use.

• v11 (this revision): the default. Hard judgments, long policies, document questions.
• v10 (revision="v10"): if you rely on sampled browser play, greedy bag-draw or slippery-grid play, knowledge multiple choice or TypeSafe-style workflow decisions, where v10 is ahead (see the losses above).

The package loads a local folder, so download the revision first:

from huggingface_hub import snapshot_download
from decider.infer import Decider
d = Decider(snapshot_download("Mapika/decider-2b", revision="v10"))

For the HTTP server, set DECIDER_MODEL to the same downloaded folder.

The decider family

All six repositories share one interface (decider.infer.Decider, POST /v1/systemone in TypeSafe's format) and one readout: the letter logits at an answer slot, softmaxed over the options. Pick by size and input.

Code, data registry, training scripts, the changelog and the per-version history: https://github.com/Mapika/decider.

Usage

from decider.infer import Decider          # decider/ is included in this repo
d = Decider("Mapika/decider-2b")
d.decide("My card was charged twice for the same purchase.",
         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
          {"question": "Does this need a refund action?", "options": ["no", "yes"]}])
# [{'choice': 'billing', 'confidence': 0.93, 'probs': {...}}, {'choice': 'yes', 'confidence': 0.70, 'probs': {...}}]   (v11, decider-ai 1.4.0, eager)

decide_batch scores many states, each with many questions, in one call. abstain_below=t returns None for decisions with confidence under t. A question can have 2 to 255 options (more than 10 options use one label token per option, see decider/prompt.py).

The same request shape as TypeSafe's Jev (POST /v1/systemone), in process or over HTTP:

d.system_one({"ticket": {"messages": [{"from": "customer", "text": "I was charged twice for order A-104. Please refund the duplicate."}]},
              "refund_policy": "Duplicate charges are eligible for a refund."},
             {"department": {"type": "choice", "instructions": "Which team should handle this?",
                             "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                          "billing": {"what": "Charges, invoices", "not_for": "delivery"}, "other": None}},
              "refund_requested": {"type": "noul", "instructions": "Does `ticket.messages[0].text` request a refund?"},
              "frustration": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": ["calm", "frustrated", "very frustrated"]}})
# {"model": "decider-2b-v11", "answers": {"department": {"type": "choice", "choice": "billing", "confidence": ..., "certainty": ..., "probabilities": {...}},
#  "refund_requested": {"type": "noul", "noul": ...}, "frustration": {"type": "score", "score": ..., "legend": {...}, ...}}, "usage": {...}}

The state may be a string, object or array (up to 32k tokens with the questions). instructions and every option description may be a string or any JSON value. Question ids are never shown to the model. Each question is scored in its own row, so an answer does not depend on which other questions are asked (independent=False packs them into one row, about half the latency for short states). Each Score level is likewise judged in its own row, without its number or its neighbours, and the per-level fits are normalised ("isolated": false restores listwise scoring). The answer also reports level_fit and their sum fit_mass, which is near 1 when exactly one level fits.

For a fixed set of questions, s = d.schema(questions) computes the question prefix once and s(state) / s.batch(states) then run only the state (1.2 to 2.4x faster per request, up to 19x per batch). It uses a questions-first prompt layout that costs accuracy: about 1.5 points on fixed label sets, 5 on per-example options, more on 50 or more options and on states of several thousand tokens. decider.serve exposes the same thing as POST /v1/systemone; the official typesafe-sdk works against it unchanged with TYPESAFE_BASE_URL pointing at the server.

Requirements: torch, transformers>=5, and flash-linear-attention (Triton kernels for the Qwen3.5 linear-attention layers; the model runs without it but several times slower). Python 3.11 or newer lets those kernels use torch.compile.

Without the helper package, the same computation in plain transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained(REPO); m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
          "Question: Which department should handle this?\nOptions:\n(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
ids = tok(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    logits = m(**ids).logits[0, -1]
letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
probs = torch.softmax(logits[letters].float() / 1.164, -1)     # -> P(billing), P(technical support), P(sales); 1.164 is the stored choice temperature

For several questions in one pass, append further Question k: ... Answer k: ( blocks and read the logits at each ( position (see decider/prompt.py).

How it works

The prompt is Context: ... followed by, for each question, the question text, the lettered options (A) ... (B) ... and an answer slot Answer k: (. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed over the valid letters, divided by the temperature in decider_config.json. Letters are never generated, so all slots are read from one pass. From decider-ai 1.4.0 the config may also hold temperature_by_type, one temperature per answer type (choice, noul, score; a missing type uses temperature). This release's config has such a map: Choice answers use 1.164, yes/no (noul) answers 1.624, Score answers 1.124 on each of their level rows. Package versions before 1.4.0 use temperature (1.145) for every answer. DECIDER_TEMPERATURE=T or Decider(path, temperature=T) replaces temperature with T and switches the map off, so every answer then uses T. Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed head.

Field types

• noul: probability of "yes".
• choice with criteria {name: description | JSON | null}: the argmax option, its probability (x_p_max, the calibrated number), confidence (TypeSafe's definition, (n·x_p_max − 1)/(n − 1) for n options), certainty (1 minus the normalised entropy) and the full distribution.
• score with criteria [level descriptions]: the expected level, the probability of the most likely level (x_p_max), confidence (TypeSafe's definition: 1 minus the expected distance from the most likely level, divided by the mean distance of the levels from the middle of the scale, floored at 0), the distribution, and the per-level fits.

Before decider-ai 1.3.0, confidence in system_one and POST /v1/systemone answers was the probability that is now x_p_max. The plain form (decide, POST /decide) still reports the top probability as confidence.

Training

Supervised stages (v1 to v8). One epoch on a mixture of public decision datasets (intent detection, ticket routing, topic classification, sentiment, emotion, moderation, NLI, paraphrase, fact verification, passage relevance, reading comprehension, multiple-choice QA, ordinal rating scales, pairwise response preference, tool selection), then continuation epochs that added next-action choice from agent trajectories (AgentGym), web element choice (Mind2Web), teacher-written situations and game states, the input shapes of the Jev API (described options, up to 255 options, JSON states with path references, long inputs), teacher- written custom questions with a generic option next to a catch-all, a second cacheable prompt layout, and isolated Score levels. In 10% of questions with three or more options an abstain option is added; in a quarter of those the option list is replaced by labels from an unrelated task so that the abstain option is correct. The full list of components with sizes is in decider/data/mixture.py of the GitHub repository; scripts/train.sh full reproduces the supervised stages in one run.

Reinforcement learning stage (v8 to v10). 384 optimizer steps at a peak learning rate of 1e-6 (cosine, 16 warm-up steps), selected among the checkpoints of a 576-step run. Each of the 48 iterations plays 4 live MiniWoB++ click tasks, 4 minesweeper boards and 4 game boards (a 5x5 grid with a slippery move, draws from bags of known composition), 4 repeats each, through the same one-pass readout that serves requests. Three loss terms use those rollouts: a PPO clipped surrogate (clip 0.2) on the terminal outcome with a leave-one-replicate-out baseline; a proper log score of the model's stated belief about the immediate outcome of its action against the exact law (games, minesweeper) or the realised outcome (browser); and a rendering-consistency term that pulls the model's answer in the other prompt layout and the reversed option order toward its served answer. A fourth term keeps the model where it was: on 8 replayed supervised rows per step, KL(v8 ‖ student) on the served distribution must stay under 0.01 nats on average and 0.05 on any row, otherwise the step drops the reward terms and follows only the KL gradient. Six browser tasks were held out from reward and used for validation only. No gold labels were used. The recipe and every measurement are in docs/RL.md of the GitHub repository.

LoRA stage (v10 to v11). A LoRA of rank 64 (alpha 128) on the attention and MLP weights of v10, trained for 2 epochs over 42,749 rows in the plain state-first layout with isolated Score levels, then merged into the bf16 weights:

The replay comes from the public mixture (scripts/train.sh full), which also holds the v9 additions (shell commands, terse routing) that v8 and v10 were not trained on; those rows were trained toward v10's answers, not their labels. No JevBench item and no Decision Index item was used for training, for writing the generators or the document questions, for selecting the checkpoint, or for the temperatures. Every training row was checked against every evaluation file used for selection, the evaluation halves of the regression tasks, the four request fixtures and the issue #9 cases. Two arms were trained: one with labels on a replay of 6,700 rows, and this one, with v10's distribution on the three times larger replay described above; the rule's fallback selected this arm after epoch 2.

Temperatures. temperature 1.145, fitted by NLL on the in-task half of the public regression set without Banking77, CLINC-OOS, MMLU, ARC, Winogrande and HellaSwag (61 tasks, 102,804 rows). temperature_by_type fitted by NLL per answer type with decider.calibrate.fit_by_type on a pool of those regression rows and our own validation rows: 108,910 Choice answers, 1,677 yes/no answers and 535 Score answers. v10's temperature (1.30) was one value fitted on the 67 in-task tasks.

Evaluation

94 public tasks, original protocol. Large label sets sub-sampled to 10 options; one temperature fitted on in-task data and stored in decider_config.json. "In-task" means the test splits of the training datasets; "held-out" means datasets never seen in training (TREC, BBC news, PAWS, SciQ, Social IQa, StrategyQA, PubMedQA, TruthfulQA, tweet irony, financial sentiment, ADE, MASSIVE scenario, student question categories, Dolly categories, CR reviews, Financial PhraseBank, CommitmentBank, QuALITY, XStoryCloze, RewardBench, Arena preferences, Hermes tool selection, and an abstention probe). ECE is the expected calibration error with 15 bins.

The two "rebuilt set" rows were measured after the data pipeline was rebuilt on another machine: two datasets no longer download (TREC-fine, the game states) and the current mixture adds held-out probes, so that set has 67 in-task and 28 held-out tasks. Its numbers are comparable to each other, not to the rows above. v10 matches v8 on it. The v10 row was measured again on 2026-09-24 in the v11 session as 0.806 / 0.474 / 0.038 and 0.755 / 0.622 / 0.084. The regression rows are all Choice answers, so v11 reads them at the Choice temperature 1.164.

Per-task accuracy / ECE on the 28 held-out datasets, v8 against v10 (not measured per task for v11)

Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against v10:

v11 against v10 on the same rows is the table under Changes from v10.

Bespoke's public suite, v10 against v11 (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through system_one as shipped; same session, v10 through decider-ai 1.3.0, v11 through 1.4.0 with the map):

Calibration of v11 on our own sets (10-bin top-label ECE on stored temperature-1 logits, read at the served temperatures; heldout_jb: held-out generated families, test_teacher2: held-out document questions, guard: knowledge guard, cal_human: human-labelled validation rows):

The map lowers the error of yes/no answers on the held-out generated families (0.171 to 0.121) and leaves Choice answers where they were (0.175 to 0.172), because the Choice temperature is fitted on a pool that is 94% everyday regression rows. A map fitted without the regression rows (Choice 1.385) would reach 0.126 there and would make the regression set less calibrated (in-task ECE 0.0510 against 0.0385); it was not used. On the JevBench hard tier, v11's top-label ECE with the map is 0.175 (0.195 at the global temperature; by type Choice 0.205, yes/no 0.243, Score 0.366). The yes/no Brier score on the probe batteries is 0.048 with the map and 0.043 at the global temperature (v10 0.046).

v10 against v8 on the same rows. Every row below is scored by both models on identical inputs and seeds. Intervals are 95% bootstrap or paired intervals.

The browser gain is in the served distribution rather than in the argmax: sampled play improves by ten points, greedy play by under one. Tic-tac-toe and minesweeper play did not change; a 2B model without search loses most of those games either way. The one measured regression is OpenJev, under one point. The JevBench row was read again for both versions on 2026-09-23, in process, bf16, decider-ai 1.2.1. The values first published here (standard / hard: v8 0.861 / 0.459, v10 0.847 / 0.459) came from the FP8 server of 2026-09-19 and do not reproduce item for item.

Bespoke's public suite (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through system_one as shipped). decider-2b v10 macro 0.704 / micro 0.711; v9 0.701 / 0.711; Nimble-9B 0.748 / 0.759; Jev 1.13.0 0.760 / 0.773 (the last two copied from Bespoke's report). Per-subset numbers, the JevBench public-item comparison (decider-2b v10 is at 1.000 / 0.889 / 0.459 on the easy / standard / hard public items, against Jev 1.13.0 at 1.000 / 0.986 / 0.730) and recordings of both versions on the same browser pages and game boards are in the GitHub README.

Speed

v11 has v10's architecture and size, and the temperature map is a division per answer, so the speed was not measured again. The numbers below were measured on v10: one NVIDIA B300, decider-ai 1.2.1, 2026-09-23. Support-ticket states of about 230 tokens with 3 typed questions each (the first 64 support_tickets examples). decider.infer.Decider uses shape-bucketed CUDA graphs; the batching server is decider/serve.py, whose default since 1.1 is bf16.

On the B300, FP8 is faster in process but slower through the server, so the server default is bf16. The schema-cache figures were measured earlier on one GH200 and not repeated: with Decider.schema, 10 described questions on short chat messages ran at 11,180 decisions/s in a batch, and one question with 151 options at 19x the full-forward rate. FP8 (e4m3 weights, per-token activation scales) changes accuracy and calibration by less than the evaluation noise.

Limitations

• v11's losses against v10 (see Changes from v10): human-labelled public sets −2.2 points, knowledge guard −1.6, greedy bag-draw play −10.9, sampled slippery-grid play −4.3, sampled browser play −2.8 (interval includes zero), TypeSafe −4.9 (interval includes zero).
• Overconfident on hard multi-step items: calibration error 0.156 on our held-out generated families and 0.175 on the JevBench public hard tier. Better than v10 (0.226, 0.307), not calibrated.
• The per-type map needs decider-ai 1.4.0 or later. With 1.3.0 or earlier, or with DECIDER_TEMPERATURE set, every answer uses 1.145: the answers are the same and yes/no answers are sharper.
• The stated-belief and click-outcome calibration of v10's RL stage (below) was not measured again on v11.
• A 2B model without reasoning. Knowledge-heavy multiple choice (MMLU, MedQA, ARC) improves little over the base model, and a judgment that needs several steps should be split into several questions.
• English only. Calibration is measured on public datasets and teacher-labelled probes, not on your traffic. Check it on your own labels before using confidence for routing.
• v10 and v11 continue the v8 weights. The v9 data for terse bucket names (support, help, account next to other) is not in them as labels (v11's replay contains those rows but trains them toward v10's answers): on held-out terse-bucket messages v8 chose the generic bucket correctly 59% of the time where v9 reached 86%. Name or describe the generic option as a bucket (general_support, or a description).
• Rules written into the question ("fill if empty, otherwise skip") are not followed at this size. State the decision as a plain question with described options.
• Picking one record out of a long JSON array by position is the least accurate input shape (0.51 with 64 records against 0.70 with one). Address records by key, or let the helper write the index into the array (0.62).
• Full label sets cost accuracy against 10 sampled options: CLINC 151-way 0.88 against 0.98; DBpedia level 2 with 70 labels is the least calibrated case (ECE 0.14).
• Questions packed into one row (independent=False) see the earlier question texts, and reversing their order changes up to 12% of answers. The default path scores each question alone.
• The browser results are on 22 click-only MiniWoB++ tasks: small synthetic pages with the elements listed as text. Typing, scrolling and real websites were not tested.
• Abstention: a catch-all option ("none of the above", "other", "unsure") is chosen when nothing on offer fits, not when the exact fine-grained label is merely absent. Wordings far from the training data remain the main risk.
• One in-task dataset, tweet_hate (SemEval-2019 HatEval), stays near chance on its test split, whose collection and label definition differ from the training split. The number is reported as measured.

Changelog

The full entries, with the browser and game recordings and the same-rows comparison against v8, are in docs/CHANGELOG.md of the GitHub repository; docs/HISTORY.md has how each stage was trained and measured.

Reproduction

Code, data registry, training and evaluation scripts, the RL recipe and the per-version history: https://github.com/Mapika/decider. Each release is staged with scripts/stage_release.py and uploaded with scripts/upload_hf.py; the previous weights are kept under the tags v10 and v8 in this repository. The LoRA stage of v11 was trained with a LoRA trainer in the research repository. eval_results.json has v11's regression metrics with the map and at the global temperature, our held-out sets by answer type, the fixtures, games, browser and Bespoke results with the paired comparisons against v10, the text games, the behaviour probes, the issue #9 cases and the JevBench public items.