od1-typed-decisions · Open Decider (4B specialist)

Open Decider is an open family of System One decision models (mvbalaji/od1-*): give a model a state (text or JSON) and typed questions (choice, noul = yes/no, score = ordinal) and it returns every answer with a full probability distribution (plus an explicit NOT_ANSWERABLE outcome), in the Jev / TypeSafe /v1/systemone shape, many questions per call.

od1-typed-decisions is the family's specialist for the typed-decisions workflows (agent-trace observability, customer service, invoice processing, security incidents). It is the Open Decider 4B general model (od1-base) further fine-tuned on the benchmark's train split, so it is a specialist in the benchmark's terms: compare it with other fitted models (e.g. Laya's typed-decisions checkpoint), not with zero-shot ones. (Renamed from od1-base-typed-decisions on 2026-09-27; the old URL redirects here.)

Results (typed-decisions test split, all 2,000 decisions)

KL and Brier use the dataset card's formulas (verified to reproduce its Uniform reference row exactly); *Jev's KL/Brier are the card's published values. Ratings: the predicted level is within one level of the gold label for 98.5% of score questions. Baselines other than Laya's native run answer each question as one choice over its options (≤ 24).

By workflow: invoice processing 0.848 · customer service 0.826 · security incidents 0.796 · agent-trace observability 0.716. Weakest questions: agent-trace urgency (0.57) and outcome (0.65), customer-service action (0.68); yes/no questions are strongest (0.79–0.96).

Calibration: the model is under-confident rather than over-confident: answers given with probability 0.6–0.7 were right 87% of the time, 0.7–0.8 → 96%, ≥ 0.8 → 100%; none of its 407 errors had confidence ≥ 0.9. Its ECE (0.157, 15 bins) therefore overstates the risk of trusting a confident answer; a temperature < 1 fitted on your own validation data sharpens the probabilities without changing any answer.

Accuracy mode (default) and the optional fast hand-off

• Default = accuracy mode: OD1Model.load(path) runs the full 4B specialist on every question (the 0.796 above).
• Optional fast modes (cascade.json, thresholds fitted on the validation split only):
  • two-model: the bundled nano/ (od1-nano-typed-decisions) answers first; the request goes to the 4B model when the Nano's lowest per-question confidence < 0.6. Test: 0.795, with 96.5% of requests escalated.
  • self-exit: the 4B model answers at its layer-8 exit and continues to full depth below confidence 0.6. Test: 0.796, 98.8% escalated.
  • On this benchmark the first stage is rarely confident enough, so the hand-off saves little time; use accuracy mode.

from od1.cascade import OD1Cascade
m = OD1Cascade.load(path, mode="two-model")   # or "self-exit"; each answer says which model answered

Correction (2026-09-27)

An earlier version of this card said the specialist was trained on hard labels. That was wrong: the training code (opendecide/losses.py::question_loss) uses cross-entropy against the benchmark's soft gold distribution whenever a record provides one, and every typed-decisions record does. A controlled re-run of the original recipe reproduced the released model's validation result exactly (0.8267, KL 0.082). The released weights, default inference and the historical test scores below are unchanged. Also corrected the same day: an earlier "~0.75 saturation line" remark, replaced by the dataset card's own definition of teacher self-agreement (see Limitations).

Input-format disclosure: four of the benchmark's six yes/no question types come with definitions of "true" and "false" (e.g. needs_review: "A human should inspect this run." / "No human attention is warranted."). This released checkpoint was trained and evaluated without those definitions in its input (the question instruction only). Our choice-wrapped competitor runs also received bare yes/no options; Laya's native run received the definitions. A corrected-input version is under evaluation and will be published separately if it is adopted.

Training

Note: 94 of the 400 test states (23.5%) have ≥ 50% of their 13-grams somewhere in the train split (template overlap; the dataset's own templated split), and 26 have ≥ 50% inside a single train state (near-duplicate example). The 0.8B specialist scored 0.776 on the 306 lower-overlap states vs 0.751 on all 400; that does not show overlap had no effect. Subset results for this model are being prepared.

Speed (one H100, bf16, batch-1 requests, CUDA graphs)

Limitations

• Fitted to the four typed-decisions workflows; outside them use the general Open Decider models (od1-base).
• The benchmark measures agreement with teacher-derived labels. The dataset card reports a teacher self-agreement of 0.735 (a fresh teacher sample scored against gold built from three other samples; "the noise floor of the labelling process"). A model fitted to the averaged gold can score above a single teacher sample; that is not by itself evidence of overfitting, nor of better real-world decisions.
• English only. Baselines were called through a choice-wrapped interface (Laya's typed-decisions checkpoint also natively), which may understate them. No frontier-LLM reference was run.

Usage

pip install torch "transformers>=5.17.0" huggingface_hub

from huggingface_hub import snapshot_download
path = snapshot_download('mvbalaji/od1-typed-decisions')
import sys; sys.path.insert(0, path)
from od1.model import OD1Model
m = OD1Model.load(path, dtype='bf16', device='cuda')   # accuracy mode; applies the fitted serving.json
m.enable_cuda_graphs()
answers = m.answer(state, {'route': {'type': 'choice', 'instructions': 'Which team?', 'criteria': {'billing': 'payments', 'tech': 'bugs'}}})

Smoke test of example.py for this checkpoint: {"route": {"answer": "billing", "confidence": 0.973}, "refund": {"answer": 0.9494752832001988, "confidence": 0.949}, "urgency": {"answer": 2, "confidence": 0.522}}

Requirements and speed

The Qwen3.5 backbone mixes full attention with gated delta-rule (linear attention) layers. Install the fast kernels, otherwise transformers silently falls back to a slow pure-PyTorch implementation of those layers and a one-question request takes hundreds of milliseconds instead of a few:

pip install "transformers>=5.17.0" flash-linear-attention==0.5.2
pip install causal-conv1d==1.7.0 --no-build-isolation   # compiles against your torch/CUDA (needs nvcc)

Check: import fla, causal_conv1d must succeed; when they are missing, transformers logs "falling back to its reference PyTorch implementation". Then enable CUDA graphs and warm up before timing:

m.enable_cuda_graphs()
for _ in range(5):                       # warm-up: kernel autotuning + graph capture per input shape
    m.answer(state, questions)
torch.cuda.synchronize(); t0 = time.perf_counter()
answers = m.answer(state, questions)
torch.cuda.synchronize(); print((time.perf_counter() - t0) * 1000, "ms")

Tested with torch 2.14.0+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, causal-conv1d 1.7.0 on one NVIDIA H100 80GB HBM3 (bf16). Expected on that setup for a short one-question request: about 35 ms without CUDA graphs and about 6 ms with them (see the speed table); other GPUs differ. CPU inference works but is slow.

Training data and licenses

The base model's training data: openly licensed datasets (see sources.csv: dataset, revision, license, split, counts), project-generated synthetic policy checks and teacher-verified structured JSON workflows (written by Qwen3.5-27B-FP8, never a compared system, without access to any benchmark data), with teacher distillation labels; plus, for this specialist, the typed-decisions train split (Apache-2.0). Held-out test sets were never trained on.

Citation

Paper in preparation (Open Decider / OpenDecider-1).