JevK5-Lite (preview, experimental)

JevK5-Lite is a CPU classifier. It reads a text and any number of label sets ("heads") in one encoder pass. It returns a calibrated probability for every label: a softmax within a single-label head, and a sigmoid per label in a multi-label head. It is a 437M-parameter DeBERTa-v3-large, fine-tuned by the JevK5 project. This is the preview-1 release; the runtime is jevk5 0.3.1.

Where it is better: calibration. On a neutral test of seven public datasets, chosen before either model was run, its probabilities are better calibrated than GLiNER2.5-Decide's. The top-label expected calibration error is lower on all six single-label sets: 0.035-0.103 against 0.091-0.250.

Where it is not: accuracy. This is a preview, and it does not beat the model it was built against.

• Clear wins: JevK5-Lite is significantly more accurate on 2 of the 7 neutral sets, AG News and Yahoo Answers.
• Within noise: it is ahead on Enron spam and tweet_eval emotion only by the 0.02 margin; the 95% intervals include zero. tweet_eval sentiment is a tie.
• Decide's wins: GLiNER2.5-Decide is clearly better on Financial PhraseBank (-0.262) and GoEmotions (-0.074), and its mean over the seven sets is higher (0.643 against 0.629).
• fast-decisions: on fastino/fast-decisions dev, GLiNER2.5-Decide is stronger: head accuracy 0.637 against 0.587, all heads right 0.488 against 0.442. It is also better calibrated there (ECE 0.137 against 0.255), so JevK5-Lite's calibration does not hold under every shift.

GLiNER2.5-Decide's training data is not public, so "neither model trained on these sets" rests on Fastino's disclosure. Both models are DeBERTa-v3-large; this card makes no speed claim.

Use

pip install "jevk5[lite] @ git+https://github.com/allebee/jevk5@v0.3.1"

from jevk5 import JevK5Lite

lite = JevK5Lite.from_pretrained("alibiserikbay/JevK5-Lite", threads=16)
lite.classify(
    "I was charged twice for the same order, please refund one of them.",
    {"intent": ["refund_request", "order_status", "cancel_subscription"],
     "areas": {"labels": ["billing", "shipping", "account"], "multi_label": True}},
)
# {'intent': {'labels': ['refund_request'], 'probabilities': {'refund_request': 0.9995, ...}},
#  'areas': {'labels': ['billing'], 'probabilities': {'billing': 0.957, 'shipping': 0.004, 'account': 0.005}}}

• Precision: dtype=torch.bfloat16 roughly halves the latency on CPUs with bf16 support (AMX, AVX512-BF16).
• Length: inputs are cut to 512 tokens, labels first, so a very long label list leaves less room for the text.
• Threshold: a multi-label head returns every label with p >= 0.5; set threshold per task to change it.

How it reads

One sequence per call: [CLS] [TASK] task (one|any) [LABEL] label ... [TASK] ... [SEP] text [SEP]. Each label is scored from the hidden state of its [LABEL] marker and the mean of the text's hidden states, by a small MLP on [label, text, label * text]. There is one calibration temperature for single-label heads (1.87) and one for multi-label heads (1.32). The design follows the label-conditioned uni-encoder idea used by GLiNER2 and GLiClass.

Evaluation

fastino/fast-decisions dev

1,700 rows, 17 domains, head accuracy averaged over domains. CPU, 16 threads. This split was used for evaluation only, once, after the model was chosen.

JevK5-Lite leads on restaurant reviews, screen tags and ticket routing. It trails most on paper field, product feedback, support intent, news topic and banking intent.

Neutral test (seven public datasets)

• Items and models: up to 500 items per set, drawn from validation or test splits. Every model got the same text, task name and the dataset's own label names. Both models ran in bf16 on a CPU with 16 threads, in one run.
• Fixed in advance: the sets, metrics and decision rule were written down before either model was run.
• Training data: none of these datasets is in JevK5-Lite's training data. GLiNER2.5-Decide's training data is not public, so "neither model trained on these sets" rests on Fastino's disclosure.
• Metric and wins: the metric is macro-F1, or exact-set accuracy for GoEmotions. A "win" means a margin of at least 0.02, or a paired bootstrap 95% CI (1,000 resamples) entirely above 0.

• Significant wins: JevK5-Lite wins AG News and Yahoo Answers; their intervals lie above zero.
• Within noise: Enron spam and tweet_eval emotion are ahead by the 0.02 margin only, with intervals that include zero. tweet_eval sentiment is a tie.
• Decide's wins: GLiNER2.5-Decide wins Financial PhraseBank and GoEmotions, and has the higher mean.
• Calibration: JevK5-Lite's top-label calibration error is lower on all six single-label sets.
• Latency: the median per set was 54-92 ms per item for JevK5-Lite and 54-97 ms for GLiNER2.5-Decide, in bf16 on 16 CPU threads.

Calibration

The temperatures were fitted on held-out synthetic data from the same generator as part of the training data. On data unlike that, the fit can be wrong in either direction. On fast-decisions dev it was overconfident (ECE 0.255). On the six single-label neutral sets it was better calibrated than GLiNER2.5-Decide.

Training

• Start: from microsoft/deberta-v3-large, one epoch over a mix of two sources.
• Public rows: 189,026 rows from public train splits: BANKING77, CLINC150, MASSIVE, HWU64, SNIPS, four Bitext assistant datasets, DBpedia-14, Civil Comments, arXiv abstracts and Schema-Guided Dialogue.
• Synthetic documents: 20,822 operational documents, repeated three times in the mix. They are support chats and emails, tickets, banking, insurance, clinic and travel requests, HR, reviews, news, sports, moderation and more. GPT-6 Luna wrote them with 1-4 label heads each, and a head was kept only when two independent solves agreed with the author.
• Label augmentation: candidate subsets, shuffled order, naming styles and task names were varied, so the model learns what labels mean rather than a fixed class list.
• Selection: on held-out slices of the public data, on held-out synthetic settings, and on held-out BANKING77/CLINC150/MASSIVE items offered with every label.
• Checks: every training row was checked against fast-decisions dev, JevBench's public items and the Jev Decision Index's test and validation sets. The check covered exact matches and shared 8-word sequences.

Limitations

• English only. It is a preview: expect uneven quality across domains, as the tables show.
• Financial sentiment and fine-grained multi-label emotion are weak.
• Calibration holds on some data and not on others (see above).
• 437M parameters, above the 400M the project aimed for.

License and notices

The weights are released under Apache-2.0. The base model, microsoft/deberta-v3-large, is MIT.

GPT-6 Luna by OpenAI wrote and checked 20,822 of JevK5-Lite's training documents through OpenAI's API. Those outputs are subject to OpenAI's terms.

Training data under their own licenses:

• BANKING77, MASSIVE and HWU64: CC BY 4.0
• CLINC150: CC BY 3.0
• SNIPS, Civil Comments and arXiv metadata: CC0 1.0
• Bitext customer support, retail banking, travel and insurance: CDLA-Sharing 1.0
• DBpedia-14: CC BY-SA 3.0
• Schema-Guided Dialogue: CC BY-SA 4.0

fastino/fast-decisions dev was used for evaluation only.

JevK5-Lite is not affiliated with Fastino or TypeSafe AI.