bert4jev

A typed-decision model built on microsoft/deberta-v3-large.

One state and any number of typed questions go in; a calibrated probability distribution per question comes back from a single forward pass. Nothing is generated, so the structured-output error rate is 0 by construction.

Three question kinds:

This repository ships float16 weights only (backbone + head), plus a small self-contained inference module (jev_infer.py). Total size: ~886 MB.

---

How it works

The input is serialized into a single sequence with three marker tokens:

[CLS] [STATE] state [Q] instructions [OPT] option_1 [OPT] option_2 ... [Q] ... [SEP]

A DeBERTa-v3-large encoder reads the state and all questions at once. For every option, a small head scores [mean(question tokens); mean(option tokens); product]; a softmax within each question's option group is that question's distribution. One forward pass answers every question.

Context: 512 tokens total, the state is truncated to 256 tokens.

---

Usage

pip install torch transformers safetensors huggingface_hub sentencepiece protobuf

from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8413, 'shipping': 0.0976, 'billing': 0.0611},
#   'confidence': 0.8413}]

Multiple questions on the same state cost one pass:

result = model.decide(
    "I was charged twice for the same order and nobody answers my emails.",
    [{"type": "choice", "instructions": "Which product area is the message about?",
      "options": ["fees & charges", "pin & security", "refund & dispute", "card", "other"]},
     {"type": "score", "instructions": "How positive is the sentiment of this message?",
      "options": ["very negative", "negative", "neutral", "positive", "very positive"]},
     {"type": "noul", "instructions": "The customer is asking for a refund."}],
)

Standard AutoModel interface

The repository ships the custom code (configuration_bert4jev.py, modeling_bert4jev.py) and registers it in config.json via auto_map. Passing trust_remote_code=True makes the standard Transformers entry points return the full decision model — no jev_infer.py needed:

from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]

AutoModel.from_pretrained(...) (without trust_remote_code) still works and returns the bare DebertaV2Model backbone, because model_type remains deberta-v2.

This path was verified to reproduce the same results as jev_infer.py: 33-case accuracy 0.879 with the same 4 errors and negation 6/6.

Standard AutoModel interface

The repository ships the custom code (configuration_bert4jev.py, modeling_bert4jev.py) and registers it in config.json via auto_map. Passing trust_remote_code=True makes the standard Transformers entry points return the full decision model — no jev_infer.py needed:

from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]

AutoModel.from_pretrained(...) (without trust_remote_code) still works and returns the bare DebertaV2Model backbone, because model_type remains deberta-v2.

This path was verified to reproduce the same results as jev_infer.py: 33-case accuracy 0.879 with the same 4 errors and negation 6/6.

Answering a structured request

The model reads a state and a set of named questions. Map every question's option set (the keys of a criteria map, or an explicit option list) to options:

import json
from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

request = {
  "state": "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "returns": "Exchanges, refunds, wrong or damaged items",
        "shipping": "Delivery status, delays, lost packages",
        "billing": "Charges, invoices, payment problems"
      }
    }
  }
}

answers = {}
for qid, q in request["questions"].items():
    options = list(q["criteria"].keys()) if "criteria" in q else list(q["options"])
    r = model.decide(request["state"],
                     [{"type": q["type"], "instructions": q["instructions"], "options": options}])[0]
    answers[qid] = {
        "type": q["type"],
        "choice": r["choice"],
        "confidence": round(r["confidence"], 4),
        "probabilities": {k: round(v, 4) for k, v in r["probabilities"].items()},
    }

print(json.dumps({"model": "bert4jev", "answers": answers}, indent=2))

---

Evaluation

All numbers below were measured on a single GPU (fp16) with the released fp16 weights, using the exact loader shipped in this repository.

Task

Customer-support routing: a free-text message must be classified into one of three departments.

{
  "returns":  "Exchanges, refunds, wrong or damaged items",
  "shipping": "Delivery status, delays, lost packages",
  "billing":  "Charges, invoices, payment problems"
}

Test sets

Results

Calibration

confidence is the maximum probability after temperature scaling (T = 1.05). Measured over the 33-case set:

Every gap is negative: the model is slightly under-confident, never over-confident, on this set. Accuracy rises monotonically with confidence.

Selective prediction

Abstaining below a confidence threshold (route to a human instead):

At threshold 0.8 the model is 100% accurate on the 52% of cases it answers.

Negation

Negation is the classic failure mode of similarity-based classifiers: "I do NOT want a refund" and "I want a refund" are topically identical. This model handles it:

Remaining errors (4 of 33)

The first three have low confidence and are caught by the abstain threshold. The fourth is a genuine multi-intent disagreement.

---

Precision

The weights are float16 (backbone and head, homogeneous dtype). Verified lossless:

Same predictions, same errors, same confidences.

---

Limitations

• Reads the question only partly; performance on unseen instructions and option sets is lower.
• English only.
• 512-token context.
• The confidence is calibrated on a validation split; re-calibrate on your own data before trusting the thresholds in production.
• It cannot generate text or arguments; it chooses among the options you give it.
• Options must be short labels or phrases. Very long or noisy option text degrades accuracy.

Files

Base model

Built on microsoft/deberta-v3-large (MIT).