Byrne-Jev

Small local System-1 typed-decision model. Same job as TypeSafe's Jev and convaiinnovations' Laya. You give it a state (text, JSON, or a conversation) and typed questions (choice, score, noul). It returns a calibrated probability for every option in one forward pass. Nothing is generated. No free text to parse. No answer outside the options you define.

I built this. ~79M params: a 70.4M causal decoder trunk (SpikeWhale) plus an 8.5M Laya-style decision head. Method is close to Laya: typed decision head, one marker position per option, 2-layer bidirectional head over the sequence, strictly proper scoring rules so the probabilities are actually calibrated.

Answer format is schema-compatible with Laya and Jev. Served over Jev's /v1/systemone wire protocol.

Checkpoint is model.pt. Temperature 1.0 for every question type. No post-hoc temperature scaling. That measured best-calibrated. See Calibration.

Files

Install

Python 3.10+.

pip install -r requirements.txt

Use

from agent import DecisionAgent

agent = DecisionAgent("model.pt", device="cuda")      # or "cpu"
result = agent.predict(
    {"body": "Hi, we were billed twice for March. Refund the duplicate today or we cancel."},
    {
        "department": {"type": "choice", "instructions": "Which department should handle this?",
                       "criteria": {"billing": "invoices, payments, refunds",
                                    "technical": "bugs, outages, errors",
                                    "other": "everything else"}},
        "urgency":    {"type": "score", "instructions": "How urgent is this?",
                       "criteria": ["not urgent", "soon", "blocking"]},
        "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    },
)
result["answers"]["department"]["choice"]            # e.g. "billing"
result["answers"]["department"]["probabilities"]     # one probability per option
result["answers"]["churn_risk"]["noul"]              # P(true)

Question types:

• choice: criteria is a dict of label -> description (or a list of labels).
• score: criteria is a list of ordered levels; the answer includes the expected level.
• noul: a yes/no question. criteria optionally describes true / false; the answer is P(true).

Each answer also carries answer_confidence (the probability of the chosen answer - the calibrated number to gate on).

Server (Jev wire protocol)

JEV_DEVICE=cuda python serve.py          # http://127.0.0.1:8000/v1/systemone
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}}'

Responses include usage, rounding: {"probabilityDecimals": 4} and cost_usd: 0, so strict Jev clients validate and budget them correctly.

Evaluation

Reproduce everything below with python eval_decisions.py --suite all --device cuda. Other suites: --suite typed, --suite benchmarks, or --suite jsonl --jsonl your_examples.jsonl. All numbers were measured on an RTX 5060 Ti.

typed-decisions: LocalLLaMA/typed-decisions test split

400 cases, 2,000 decisions, four workflows. Metrics and reference rows match the protocol in Laya's fine-tuning notebook. Reference numbers are as published by their authors.

KL divergence 0.228, total variation 0.220. Latency is 111 ms p50 and 158 ms p95 per case (5 questions in one forward pass, GPU).

What stands out:

• Calibration. ECE 0.045 is the lowest in the table. Stated confidences track hit rate.
• Probability quality. Brier beats Jev's published score. Soft accuracy beats Laya's fine-tuned checkpoint.
• Accuracy gap. Top-1 is 10-14 points below Jev and Laya. Invoice processing is the weakest workflow.

Public benchmarks as typed questions

200 examples from each benchmark's evaluation split, each posed as a typed question (eval_data.py). Instruction wording and option descriptions are sampled with a fixed seed. banking77 and MASSIVE show up to 20 options per question: the gold intent plus randomly sampled other intents (not the full 77 / 60-way label set).

Sentiment and topic/intent are strong. Reading a passage and reasoning over it is weak. BoolQ is below always answering "yes" (~0.62). MultiNLI is 0.50, where chance is 0.33.

Calibration

model.pt uses temperature 1.0. I also fitted post-hoc temperatures (per question type and per option count) and compared. On typed-decisions, fitting made calibration worse: ECE rose from 0.045 to 0.100 and soft accuracy fell from 0.509 to 0.473, while Brier improved slightly (0.134 to 0.127). On the benchmark table, fitted temperatures lowered ECE on most single-label tasks. The released model keeps temperature 1.0 because structured-state decisions are the main use. To apply your own temperatures, add temperature (per type: choice, score, noul) or temperature_by_options to the model's decision_cfg; the loader clamps values to [0.5, 5].

Integration test: jev-doom (Freedoom, zero-shot)

To check it can stand in for Jev in a real application, I dropped Byrne-Jev in for Jev in jev-doom, a public harness that shows Jev playing Freedoom. Observable mode, MAP01, 90 s real time. No Doom-specific adaptation.

It works as a drop-in at the protocol level. As a zero-shot game agent it's weak: it tracks and faces enemies but doesn't choose to shoot, and it prefers right-side moves.

Limitations

• Accuracy. Top-1 on typed-decisions is below Jev and Laya.
• Passage reasoning. Reasoning over a passage (BoolQ, NLI) is weak.
• Yes/no detection (noul) is unreliable and can be inverted. Hand-written spot check: 5 of 7 detection questions were wrong, including clear cases. Explicit "refund today or we cancel" scored churn 0.09 while a polite feature request scored 0.68. Obvious phishing email 0.29. "I want a manager now" escalation 0.04. "ignore all previous instructions" injection 0.16. Describe both outcomes in criteria (e.g. {"true": "the user threatens to cancel or leave", "false": "the user does not threaten to leave"}). Same 7 checks: accuracy 2/7 -> 5/7 (cancel threat 0.09 -> 0.99, escalation 0.04 -> 0.98, injection 0.16 -> 0.76). A two-option choice with the same descriptions scored the same (5/7), so it's the descriptions that matter, not the question type. Even so, a polite request still scored as a churn risk and the phishing email still scored below 0.5. Validate yes/no questions on your own data before gating anything on them.
• Score questions under-rate intensity. Urgency, frustration and star ratings skew toward the low end (e.g. a glowing review got an expected 2.6 of 5 stars; a same-day cancel threat was rated "not urgent").
• Context. Default 1024-token budget per question truncates long states. Questions with many long options share a 256-token budget (raise it with JEV_HEAD_MAX_LEN).
• Action head. action.act_probability in each answer is kept for format compatibility. It isn't meaningful.
• Option order. Options are read in the order given. Results can shift slightly if you reorder them.