DecidaBERT-large-v2
A one-forward-pass decision model with an 8,192-token context. It reads a piece of state and a set of typed questions and returns a probability distribution for every question at once. No tokens are generated.
It takes the POST /v1/systemone request shape: noul (yes or no), choice (one of N labels) and score (an
ordered rubric). It is the successor of DecidaBERT-large.
Results
Typed Decisions test split (400 cases, 2,000 questions), scored as for DecidaBERT-large
A prediction is right when its top answer equals the gold label. Intervals are 95% Wilson intervals. Both models
were fitted on the benchmark's train split (specialist mode); the test split was never used for training,
calibration or choosing a checkpoint.
| question type | questions | DecidaBERT-large | DecidaBERT-large-v2 | v2 95% interval | random guess |
|---|---|---|---|---|---|
| choice | 600 | 0.722 | 0.747 | 0.710 to 0.780 | 0.233 |
| score | 800 | 0.694 | 0.759 | 0.728 to 0.787 | 0.244 |
yes/no (noul) |
600 | 0.830 | 0.868 | 0.839 to 0.893 | 0.500 |
| all | 2,000 | 0.743 (0.723 to 0.762) | 0.788 | 0.770 to 0.805 |
For score questions v2 is within one level of the gold level 97.6% of the time (v1: 95.5%).
Benchmark metrics (the benchmark's own scorer)
| Accuracy ↑ | KL from gold ↓ | Brier ↓ | ECE ↓ |
|---|---|---|---|
| 0.7835 | 0.0861 | 0.0473 | 0.1435 |
The two accuracy figures differ only in how ties are read: the benchmark's scorer takes the most likely option of
the gold distribution, the table above takes the gold label. The test predictions are in
eval/typed-decisions/predictions.jsonl so both can be re-derived.
Confidence
probabilities is the raw distribution (the one scored above): it follows the gold distributions closely, which
are soft, so the top answer's probability reads low (mean 0.64 while the top answer is right 79% of the time).
Every answer therefore also carries a confidence: the top answer's probability after a per-type temperature
fitted on held-out train cases. On the test split its calibration error is 0.023 (raw: 0.144), so a
confidence of 0.8 means right about 80% of the time; use it for thresholds and routing.
How it works
- Backbone: ModernBERT-large (395M parameters, 28 layers), bidirectional, context 8,192 tokens (v1: trained at 512, up to 2,048).
- One pass for all questions: the state is read once; every question and its options follow it in the same sequence, so a request with many questions costs one forward pass, not one per question.
- Readout: a marker token per option (and one per yes/no question); three small heads (yes/no, choice, score) map the marker states to scores; a softmax (or sigmoid) gives the distribution.
- Training data: decision records from multiple domains of IT and software work (below), balanced by design,
then the
trainsplit of LocalLLaMA/typed-decisions. A targeted weight edit was applied before the last stage. No test split of any benchmark was used for training.
| Domain group | Records | Tasks |
|---|---|---|
| Gating and classification | 6,834 | 11 |
| Routing and selection | 6,834 | 9 |
| Agent control flow (go / no-go) | 6,833 | 9 |
| Ranking and scoring | 6,833 | 8 |
| Verification | 6,833 | 9 |
| Operational engineering | 6,833 | 14 |
| Total | 41,000 | 60 |
LocalLLaMA/typed-decisions train |
1,000 | 4 workflows |
Coverage: 38 roles (developers, SREs, SOC analysts, IT technicians, architects and more), 48 settings and 24 formats (logs, diffs, alerts, tickets, agent traces, configs, metrics tables and more); 165,952 questions, of which 44% yes/no, 30% choice and 26% score.
Files
| File | What it is |
|---|---|
model.safetensors, config.json |
encoder and decision heads (FP32), encoder config |
decision_config.json |
context and token budgets, confidence temperatures |
tokenizer.json, tokenizer_config.json |
tokenizer |
decidabert_v2/ |
inference code |
decision_server.py |
a /v1/systemone server |
eval/typed-decisions/predictions.jsonl |
test-split predictions |
Usage
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("helmo/DecidaBERT-large-v2")
sys.path.insert(0, path)
from decidabert_v2 import load
model = load(path)
answers = model.predict(
"Alert: 41 failed SSH logins for root from one external IP in 3 minutes, then one success.",
{
"disposition": {"type": "choice", "instructions": "How should the alert be handled?",
"criteria": {"close": "Benign, close it.", "investigate": "Open an investigation.",
"contain": "Isolate the host now."}},
"true_positive": {"type": "noul", "instructions": "Is this a real attack?"},
"severity": {"type": "score", "instructions": "How severe is it?", "criteria": ["Low", "Medium", "High", "Critical"]},
},
)
/v1/systemone server
pip install torch transformers safetensors fastapi uvicorn huggingface_hub
huggingface-cli download helmo/DecidaBERT-large-v2 --local-dir DecidaBERT-large-v2
cd DecidaBERT-large-v2 && PORT=8000 python decision_server.py
Limits
- English only. Best when the state describes the situation in words; raw numbers are read less reliably.
- Fitted on the benchmark's
trainsplit: its numbers there belong with other fitted models, not zero-shot ones. - The gold labels of the benchmark are a teacher model's averaged answers, so scores measure agreement with that teacher.