od1-nano-v2-typed-decisions · Open Decider (0.8B specialist)

Open Decider is an open family of System One decision models (mvbalaji/od1-*): give a model a state (text or JSON) and typed questions (choice, noul = yes/no, score = ordinal) and it returns every answer with a full probability distribution, many questions per call.

od1-nano-v2-typed-decisions is the family's small (0.8B) specialist for the typed-decisions workflows. It is a new version of od1-nano-typed-decisions (which stays available, unchanged): it starts from the stronger Nano v2 general model instead of Nano v1. It was fitted on the benchmark's train split, so it is a specialist: compare it with other fitted models, not zero-shot ones. For the highest accuracy use the 4B od1-typed-decisions.

Results

typed-decisions official test split (400 cases / 2,000 decisions; this test split has been used to evaluate other models in our project, so treat comparisons on it as exploratory):

Paired difference v2 − v1: +1.4 pt, 95% interval [−0.25, +3.0] (bootstrap over cases) -- the improvement on this test is not statistically established, and the interval includes a small regression. By workflow (v2 / v1): agent traces 0.702 / 0.674, customer service 0.800 / 0.762, invoices 0.806 / 0.812, security 0.752 / 0.756. Ratings: MAE of the expected level 0.409 (v1 0.447), within one level 0.979. KL and Brier use the dataset card's formulas.

Overlap subsets (supporting information): template-overlap cases (94) 0.698 (v1 0.668), lower-overlap cases (306) 0.786 (v1 0.777), near-duplicate cases (26) 0.746 (v1 0.715).

Secondary evaluation on four new workflows (KYC onboarding, field service, inventory discrepancy, adverse-event reports; 236 cases / 1,180 decisions): 832 / 1,180 = 0.705 vs 544 / 1,180 = 0.461 for v1 (+24.4 pt, 95% interval [+21.4, +27.4]); the 4B specialist scores 0.777 there. These cases were written and labelled by the same teacher model (Qwen3.5-27B-FP8) that labelled our general training data, were not human-reviewed, and had already been opened for an earlier model comparison: this is agreement with a familiar teacher on new workflows, not an untouched, independently verified confirmation.

Speed and memory (measured)

One idle H100 80GB, bf16, CUDA graphs, batch-1 requests, one system per process, end-to-end answer() time after warm-up, p50 (p95) ms per request; states of 9–36 tokens (short) and 201–398 tokens (medium):

With CUDA graphs enabled, the first request with a new input shape is slower (short states: ~210–280 ms p50). Without CUDA graphs (same workload), warm p50 is 34.8 / 35.5 / 41.9 ms for 1 / 5 / 20 questions on short states and 35.9 / 43.1 / 92.3 ms on medium states, and first calls take ~70–85 ms.

Memory. The weights take 1.5 GB. In our benchmark, peak GPU memory was approximately 16.6 GB with CUDA graphs both enabled and disabled. The cause of the peak has not been isolated. This is workload-specific peak memory, not weights-only memory or a universal serving requirement. (Measurement: torch.cuda.max_memory_allocated over all cold, warm-up and timed requests of the workload above -- six short and six medium states, 1, 5 and 20 questions per request, batch 1, one request at a time -- in a process running only this model.)

Correction (2026-09-27): an earlier version of this card attributed the peak mostly to CUDA-graph memory pools; a measurement with CUDA graphs disabled showed the same peak, so that attribution was removed.

Training

Limitations

• Fitted to the four typed-decisions workflows; on new workflows accuracy drops (0.705 on the teacher-labelled set above).
• The benchmark measures agreement with teacher-derived labels; the dataset card reports a teacher self-agreement of 0.735 under its protocol. Scores are not independently verified decision quality.
• English only; no unanswerable cases were evaluated, so no rejection claims.

Usage

pip install torch "transformers>=5.17.0" huggingface_hub

from huggingface_hub import snapshot_download
path = snapshot_download('mvbalaji/od1-nano-v2-typed-decisions')
import sys; sys.path.insert(0, path)
from od1.model import OD1Model
m = OD1Model.load(path, dtype='bf16', device='cuda')
m.enable_cuda_graphs()
answers = m.answer(state, questions)   # typed-decisions style: {qid: {"type": ..., "instructions": ..., "criteria": ...}}

Requirements and speed

The Qwen3.5 backbone mixes full attention with gated delta-rule (linear attention) layers. Install the fast kernels, otherwise transformers silently falls back to a slow pure-PyTorch implementation of those layers and a one-question request takes hundreds of milliseconds instead of a few:

pip install "transformers>=5.17.0" flash-linear-attention==0.5.2
pip install causal-conv1d==1.7.0 --no-build-isolation   # compiles against your torch/CUDA (needs nvcc)

Check: import fla, causal_conv1d must succeed; when they are missing, transformers logs "falling back to its reference PyTorch implementation". Then enable CUDA graphs and warm up before timing:

m.enable_cuda_graphs()
for _ in range(5):                       # warm-up: kernel autotuning + graph capture per input shape
    m.answer(state, questions)
torch.cuda.synchronize(); t0 = time.perf_counter()
answers = m.answer(state, questions)
torch.cuda.synchronize(); print((time.perf_counter() - t0) * 1000, "ms")

Tested with torch 2.14.0+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, causal-conv1d 1.7.0 on one NVIDIA H100 80GB HBM3 (bf16). Expected on that setup for a short one-question request: about 35 ms without CUDA graphs and about 6 ms with them (see the speed table); other GPUs differ. CPU inference works but is slow.

Training data and licenses

The Nano v2 base model's training data: openly licensed datasets (see sources.csv), project-generated synthetic policy checks and teacher-verified structured JSON workflows (written by Qwen3.5-27B-FP8, never a compared system), with teacher distillation labels; plus, for this specialist, the typed-decisions train split (Apache-2.0).

Citation

Paper in preparation (Open Decider / OpenDecider-1).