Nano-Jev v1.0

Version 1.0 (stable). Jev-style decision model. Independent; not affiliated with TypeSafe AI or the NanoJev GitHub project.

A small (33.4M parameter) typed decision model for RAG pipelines. Give it a question, a set of options and some context; it returns a calibrated probability for each option in a few milliseconds. It never generates text.

• Package: pip install nano-jev · Code: github.com/shubham10divakar/nano-jev
• All versions stay available on this repo as git tags: v1.0 (this, default) and v0.1.

v1.0 specs

Versions

Usage

pip install nano-jev

import nanojev

d = nanojev.load()          # v1.0 (default); nanojev.load("v0.1") for the older version

d.relevance("When was UCL founded?", ["University College London was founded in 1826."])
# [{'irrelevant': 0.15, 'partially relevant': 0.17, 'directly answers': 0.68}]
d.sufficient(query, passages)   # {'yes': .., 'no': ..}
d.grounded(claim, context)      # {'yes': .., 'no': ..}

Command line: nano-jev list, nano-jev relevance -q QUERY -p PASSAGE, and more; see the README.

With plain transformers (no extra code):

import json, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo, rev = "sdmlai/nano-jev", "v1.0"
tok = AutoTokenizer.from_pretrained(repo, revision=rev)
model = AutoModelForSequenceClassification.from_pretrained(repo, revision=rev).eval()
temps = json.load(open(hf_hub_download(repo, "calibration.json", revision=rev)))

query, passage = "When was UCL founded?", "University College London was founded in 1826."
question = f"How relevant is this passage to the query: {query}"
options = ["irrelevant", "partially relevant", "directly answers"]
enc = tok([f"question: {question} option: {o}" for o in options], [passage] * len(options),
          truncation="only_second", max_length=512, padding=True, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits.squeeze(-1)
print(dict(zip(options, torch.softmax(logits / temps["relevance"], dim=0).tolist())))

Question templates: relevance How relevant is this passage to the query: {query}; sufficient Does the context contain enough information to answer: {query} (context = passages as [1] title: text lines); grounded Is this claim supported by the context? Claim: {claim}.

Results

Temperatures and baseline mappings are fitted on the in-distribution calibration split and applied unchanged everywhere.

Test split (held-out halves of HotpotQA / SQuAD 2.0 / MultiNLI validation)

Calibration error (ECE) after temperature scaling: relevance 0.017, sufficient 0.024, grounded 0.048.

Held-out datasets (never used for training or calibration)

MuSiQue (multi-hop QA with hard retrieved distractors) for relevance and 2-hop sufficiency; VitaminC (claim verification) for groundedness.

Accuracy drops by about 15–20 points on unseen datasets, and calibration does not transfer (held-out ECE 0.17–0.18). Re-fit the temperatures, or at least re-check your thresholds, on your own data.

Speed

ms per decision, batched, RTX 3060: v1.0 1.0–4.0 · v0.1 0.6–2.3 · bge-reranker 3.5–17.9 · Qwen3-4B 54.5–379.1. On CPU (Ryzen 7 5800X, 8 threads) v1.0 takes ~17 ms per decision and v0.1 ~9 ms.

Cascade: escalate unsure grounded decisions to an LLM

Accept Nano-Jev's verdict when its top probability ≥ a threshold, otherwise ask Qwen3-4B-Instruct. The threshold (0.80) was chosen on the calibration split.

On the test split the cascade beats both models alone while sending only a quarter of the decisions to the LLM. On held-out data a 0.90 threshold matches the LLM alone with about half the LLM calls. Full sweep and plot: results/v1.0/cascade.

def grounded(claim, context, threshold=0.8):
    p = d.grounded(claim, context)
    if max(p.values()) >= threshold:
        return p                       # confident: ~1 ms
    return ask_llm(claim, context)     # unsure: escalate

Limitations

• Out-of-distribution accuracy is much lower than on the test split (see above), and calibration fitted in-distribution is overconfident on new data.
• Custom option sets (decide() with your own labels) are weaker than in v0.1, whose MS MARCO-trained base had learned general text–label matching. For custom options, consider v0.1 (mind its licence note) or fine-tune on your labels.
• English, Wikipedia-style text.
• "partially relevant" follows HotpotQA rules: a supporting paragraph without the answer string.

Licence

Weights: MIT, the same licence as the base model; no MS MARCO lineage.

Fine-tuning data: HotpotQA (CC BY-SA 4.0), SQuAD 2.0 (CC BY-SA 4.0), MultiNLI (see its licence terms). Held-out evaluation: MuSiQue (CC BY 4.0), VitaminC (CC BY-SA 3.0).