DecidaBERT-large-v1.1
A one-forward-pass decision model with an 8,192-token context. It reads a piece of state and a set of typed questions and returns a probability distribution for every question at once. No tokens are generated.
It takes the POST /v1/systemone request shape: noul (yes or no), choice (one of N labels) and score (an
ordered rubric). It improves on DecidaBERT-large (v1) and is a
sibling of DecidaBERT-large-v2: same architecture and request
shape, the same accuracy overall within noise, a different base and training recipe, and different strengths
(below). If you
want a single model, start with v2; try v1.1 where its strengths match your work, or run both.
Results
Typed Decisions test split (400 cases, 2,000 questions), scored as for DecidaBERT-large
A prediction is right when its top answer equals the gold label. Intervals are 95% Wilson intervals. All three
models were fitted on the benchmark's train split (specialist mode); the test split was never used for
training, calibration or choosing a checkpoint. These weights were scored on it once. (The run was scored once
before, at 0.780 overall: its final weights were then lost to a storage failure and re-created by resuming the run
from its last checkpoint; the test result did not inform anything.)
| question type | questions | DecidaBERT-large (v1) | DecidaBERT-large-v2 | DecidaBERT-large-v1.1 | v1.1 95% interval | random guess |
|---|---|---|---|---|---|---|
| choice | 600 | 0.722 | 0.747 | 0.747 | 0.710 to 0.780 | 0.233 |
| score | 800 | 0.694 | 0.759 | 0.755 | 0.724 to 0.784 | 0.244 |
yes/no (noul) |
600 | 0.830 | 0.868 | 0.850 | 0.819 to 0.876 | 0.500 |
| all | 2,000 | 0.743 (0.723 to 0.762) | 0.788 (0.770 to 0.805) | 0.781 | 0.762 to 0.799 |
For score questions v1.1 is within one level of the gold level 98.3% of the time (v2: 97.6%, v1: 95.5%).
v1.1 against v2, question by question
Both models answered the same 2,000 questions. The difference is v1.1 minus v2; its interval resamples whole cases (a case's questions share one state); p is an exact McNemar test on the questions only one model got right.
| v2 | v1.1 | difference (95% interval) | only v2 right | only v1.1 right | p | |
|---|---|---|---|---|---|---|
| all | 0.788 | 0.781 | −0.007 (−0.021 to +0.008) | 99 | 85 | 0.34 |
| choice | 0.747 | 0.747 | 0.000 (−0.024 to +0.025) | 26 | 26 | 1.00 |
| score | 0.759 | 0.755 | −0.004 (−0.026 to +0.019) | 39 | 36 | 0.82 |
| yes/no | 0.868 | 0.850 | −0.018 (−0.044 to +0.008) | 34 | 23 | 0.19 |
| agent trace observability | 0.730 | 0.732 | +0.002 | 11 | 12 | 1.00 |
| customer service | 0.830 | 0.810 | −0.020 | 29 | 19 | 0.19 |
| invoice processing | 0.806 | 0.832 | +0.026 | 18 | 31 | 0.09 |
| security incidents | 0.786 | 0.750 | −0.036 | 41 | 23 | 0.03 |
Overall the two are level. v1.1 leans ahead on invoice processing and behind on security incidents and yes/no questions; only the security gap reaches p < 0.05, and with seven subgroups compared it may be chance. The two give the same top answer on 90% of the questions.
Benchmark metrics (the benchmark's own scorer)
| model | Accuracy ↑ | KL from gold ↓ | Brier ↓ | ECE ↓ |
|---|---|---|---|---|
| DecidaBERT-large-v2 | 0.7835 | 0.0861 | 0.0473 | 0.1435 |
| DecidaBERT-large-v1.1 | 0.776 | 0.0852 | 0.0475 | 0.150 |
The two accuracy figures differ only in how ties are read: the benchmark's scorer takes the most likely option of
the gold distribution, the tables above take the gold label. The test predictions are in
eval/typed-decisions/predictions.jsonl so both can be re-derived.
Confidence
probabilities is the raw distribution (the one scored above): it follows the gold distributions closely, which
are soft, so the top answer's probability reads low. Every answer therefore also carries a confidence: the top
answer's probability after a per-type temperature fitted on 200 held-out train cases. On the test split its
calibration error is 0.024 (raw: 0.155), so a confidence of 0.8 means right about 80% of the time; use it for
thresholds and routing.
How it works
- Backbone: ModernBERT-large-nli, a ModernBERT-large further trained on natural language inference (395M parameters, 28 layers), bidirectional, context 8,192 tokens.
- One pass for all questions: the state is read once; every question and its options follow it in the same sequence, so a request with many questions costs one forward pass, not one per question.
- Readout: a marker token per option (and one per yes/no question); three small heads (yes/no, choice, score) map the marker states to scores; a softmax (or sigmoid) gives the distribution.
- Training: one run of two epochs over the whole mix at once (v2 trained in stages, with a targeted weight edit
and a final fit on the benchmark's
trainsplit). The checkpoint was chosen on the 200 held-outtraincases. (The last 227 of its 2,854 steps were re-run from the run's own checkpoint after the final weights were lost to a storage failure.) No test split of any benchmark was used for training, and training states that match a benchmark test or held-out state were removed.
Training data includes:
| Data | Records | Questions | License |
|---|---|---|---|
| Decision records from multiple domains of IT and software work, as for v2 (6 domain groups, 60 tasks, 38 roles) | 41,000 | 165,952 | |
LocalLLaMA/typed-decisions train (1,000 of its 1,200 cases, repeated 5 times) |
1,000 | 5,000 | Apache-2.0 |
| helmo/synthetic-typed-decisions | 9,879 | 9,879 | MIT |
| frontier-infra/jebadiah-synth-v2 | 3,253 | 14,714 | Apache-2.0 |
kgrozdanovski/assistant-decisions (train, long/train) |
5,101 | 10,619 | Apache-2.0 |
Files
| File | What it is |
|---|---|
model.safetensors, config.json |
encoder and decision heads (FP32), encoder config |
decision_config.json |
context and token budgets, confidence temperatures |
tokenizer.json, tokenizer_config.json |
tokenizer |
decidabert_v1_1/ |
inference code |
decision_server.py |
a /v1/systemone server |
eval/typed-decisions/predictions.jsonl |
test-split predictions |
Usage
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("helmo/DecidaBERT-large-v1.1")
sys.path.insert(0, path)
from decidabert_v1_1 import load
model = load(path)
answers = model.predict(
"Alert: 41 failed SSH logins for root from one external IP in 3 minutes, then one success.",
{
"disposition": {"type": "choice", "instructions": "How should the alert be handled?",
"criteria": {"close": "Benign, close it.", "investigate": "Open an investigation.",
"contain": "Isolate the host now."}},
"true_positive": {"type": "noul", "instructions": "Is this a real attack?"},
"severity": {"type": "score", "instructions": "How severe is it?", "criteria": ["Low", "Medium", "High", "Critical"]},
},
)
Each answer carries its result (choice: the top label; score: the expected level; noul: the probability of
yes), probabilities for choice and score questions, and confidence.
/v1/systemone server
pip install torch transformers safetensors fastapi uvicorn huggingface_hub
huggingface-cli download helmo/DecidaBERT-large-v1.1 --local-dir DecidaBERT-large-v1.1
cd DecidaBERT-large-v1.1 && PORT=8000 python decision_server.py
Limits
- English only. Best when the state describes the situation in words; raw numbers are read less reliably.
- Fitted on the benchmark's
trainsplit: its numbers there belong with other fitted models, not zero-shot ones. - The gold labels of the benchmark are a teacher model's averaged answers, so scores measure agreement with that teacher.
- On the benchmark it is weaker than v2 on security incidents and yes/no questions.