BaseDecision-Dyno

A 4B-class decision model with calibrated option probabilities.

Give BaseDecision-Dyno a document, conversation, policy, or structured record—and questions with caller-defined options. It returns the selected option and a probability for every option. It scores decisions directly, without generating explanations or decoding an answer sequence.

The calibrated BF16 checkpoint scores 54.80 on the full public Decision Index 0.3 suite, using the official evaluation kit.

At a glance

Quickstart

The release includes custom model code. Review it before enabling trust_remote_code=True. Use AutoModel and the structured decide() interface to preserve the trained prompt, attention, option mapping, and calibration.

The validated environment used Python 3.12.15, PyTorch 2.14.0+cu130, and Transformers 5.17.0. Package loading and sampled protocol checks also passed on an NVIDIA A100-SXM4-80GB. The examples below require a compatible NVIDIA CUDA GPU.

import torch
from transformers import AutoModel
MODEL_PATH = "onlyaady/BaseDecision-Dyno"
# Match the backend setting used by the evaluated engine.
torch.backends.cuda.enable_cudnn_sdp(False)
model = AutoModel.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": "cuda:0"},
).eval()
request = {
    "state": "The customer's invoice contains the same charge twice.",
    "questions": {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "billing": "Charges, invoices, and refunds",
                "technical": "Technical integration problems",
                "sales": "Questions about buying the product",
            },
        },
        "duplicate": {
            "type": "noul",
            "instructions": "The customer was charged twice for the same item.",
        },
    },
}
# Read twice is the default. Calibration is applied internally.
result = model.decide(request)
assert result["read_twice"] is True
print(result["answers"]["team"]["choice"])
print(result["answers"]["team"]["probabilities"])
print(result["answers"]["duplicate"]["probabilities"]["true"])
# Explicit single-read mode, with its own calibration applied internally.
flash_result = model.decide(request, mode="flash")
assert flash_result["read_twice"] is False

For a single question, {"state": ..., "question": ...} is also accepted; its answer is returned under answers["q"]. Multiple questions are processed independently, each in its own sequence and forward pass. They do not share a single prefill pass.

The response includes answers, mode, read_twice, sequence_tokens, calibrated, and release_id. For both decision types, the native API returns choice and probabilities. A noul answer uses the probability keys "true" and "false"; the Decision Index adapter converts this distribution into the benchmark’s scalar noul response.

Persistent inference

Download the model and its included JSONL runner:

hf download onlyaady/BaseDecision-Dyno --local-dir BaseDecision-Dyno
cd BaseDecision-Dyno
# Default: read twice, with calibration applied internally.
python inference.py --input requests.jsonl
# Explicit single-read mode, with its own calibration.
python inference.py --mode flash --input requests.jsonl

Each input line is one structured JSON request; each output line is one JSON response. Create requests.jsonl with your requests before running these commands.

The runner loads the model once and keeps it resident across successive requests. Omit --input to read requests from stdin until EOF. Use the validated CUDA environment. No compilation, quantization, or cross-request batching is enabled by this runner.

Read twice by default

Read twice repeats the state and question/options content within one rendered sequence, followed by one scoring pass per question. It is not two independent predictions, voting, or generated reasoning.

• model.decide(request), model(request), and model.prepare_request(request) default to read twice.
• mode="flash" explicitly selects a single read.
• The 16,384-token limit applies after rendering and repetition. Default mode therefore accepts less original content than flash mode.
• Oversized requests raise a length error. Inputs are not silently truncated, and default mode never falls back automatically to flash.
• Keep state and questions structured; do not manually repeat the input before calling decide().

The default is implemented in the supplied model code rather than left to the caller's prompt. Someone who modifies the code or directly uses lower-level weights can change that behavior.

Decision Index results

Completed self-run public evaluation with the official kit, edition 0.3. The exact calibrated model and strict default protocol above were used. These are public results, not a maintainer-run private or Full leaderboard score.

Index values are scores, not accuracy percentages. The public composite includes 37 panel benchmarks. The official report also contains additional diagnostic benchmark entries.

Raw scores combine benchmark-specific metrics; they should not all be read as plain accuracy. See the accompanying scores.json for individual benchmarks, coverage, and scoring details.

Unsupported requests were retained in the official scoring path: POP909-CL (158), ToolRet (2), API-Bank (1), and HLE (1). The summary reports alone do not establish the individual refusal reason for every row.

Calibration

Mode-specific scalar temperatures are applied once to raw logits, followed by a softmax over the current question's options:

probabilities = softmax(raw_logits / temperature)

decide() already applies the appropriate temperature. Do not apply another temperature to its returned probabilities. These values replace the earlier 1.26 setting.

Temperatures were fitted on 10,000 decisions and audited on 10,000 separate decisions: 8,000 held out by group and 2,000 from families absent from temperature fitting. Those families were not necessarily absent from backbone training.

Accuracy on the 9,266 hard-labeled audit decisions was unchanged by calibration: 70.26% in default mode and 69.78% in flash mode. Positive scalar temperature scaling preserves the selected option. These audit accuracies are separate from the Decision Index score.

Calibration improved on this audit distribution; it is not guaranteed on every domain. Training, quantization, or a material backend change requires renewed evaluation and may require recalibration.

Performance and deployment

The completed public run measured the following request latencies on shared NVIDIA RTX PRO 6000 Blackwell resources:

These are CUDA-synchronized individual-request timings from the public evaluation. Shared physical GPU contention and the benchmark request mix affect the measurements. They are not the maintainers' dedicated-hardware latency result, and they exclude model loading and external network/queueing overhead.

On an earlier short-input sample (60 distinct requests per mode, each measured three times; up to 512 original single-read rendered tokens and at most 32 options), the uncompiled runtime measured:

The short-input results do not establish a universal 150 ms default or 100 ms flash bound. Long contexts and many-question requests can take substantially longer.

Keep the model loaded across requests. The supplied runner does this without changing the scoring path. Compilation and quantized inference are not qualified for this release. A 16GB laptop deployment remains a target; this BF16 GPU release does not establish laptop memory fit or latency.

Limitations

• Text-only release; image, audio, and video inference are not supported.
• The evaluated adapter supports choice and noul; a dedicated ordinal score response interface is not qualified by this release.
• Default repetition consumes context and compute. The context limit is per rendered question, not 16k original tokens plus a second copy.
• Option probabilities are normalized over the supplied options. They do not prove that the option set contains a correct answer.
• English-focused evaluation does not establish comparable multilingual performance.
• Factual, numerical, ambiguous, and out-of-domain decisions can be wrong, including when confidence is high.
• Adversarial robustness and resistance to instructions planted in the state have not been established.
• Owners can modify the supplied runtime; read-twice enforcement is an API behavior, not a cryptographic restriction on weights.

Licensing and acknowledgements

The BaseDecision-Dyno software repository & model weights are licensed under Apache-2.0.

This release builds on Qwen3.5-4B; Qwen3.5-4B's base license does not by itself settle the licensing of this derived release or all training sources. Preserve applicable upstream notices and dataset terms.

Special Thanks to ezjev & Maintainer of tasksource.