Classify, rate and answer yes / no about any text or JSON — with calibrated probabilities, in one pass, no GPU, no
text generation. Drop-in compatible with the Laya / Jev question format and /v1/systemone API.
Measured on a 2014 dual-Xeon workstation with no GPU, using Laya's own latency workload. Laya re-reads the text for every question; Nirnay reads it once — so 10 questions take 0.23 s instead of 1.4 s, and 50 take 0.65 s instead of 6.7 s.
Try it free in Colab (CPU runtime): nirnay_demo.ipynb · Code: github.com/nandukmelath/nirnay
nirnay.load("nandukmelath/nirnay", subfolder="v2").answer_confidence is a probability you can gate on; per-type temperatures ship with the model.learn() with 16 examples per label takes Banking77 from 0.598 to
0.840 on a CPU.pip install git+https://github.com/nandukmelath/nirnay
# or: pip install https://huggingface.co/nandukmelath/nirnay/resolve/main/dist/nirnay-1.0.0-py3-none-any.whl
import nirnay
m = nirnay.load("nandukmelath/nirnay") # v3.1, most accurate (default)
# m = nirnay.load("nandukmelath/nirnay", subfolder="fast") # v1, fastest
# m = nirnay.load("nandukmelath/nirnay", subfolder="v2") # the previous default, pinned
r = m.predict(
{"ticket": {"subject": "Payout failing", "body": "My payouts failed for 3 days, please help ASAP"}},
{"team": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}},
"urgent": {"type": "noul", "instructions": "Is this ticket urgent?"},
"severity": {"type": "score", "instructions": "How severe is the problem?",
"criteria": ["minor", "moderate", "severe"]}})
r["answers"]["team"]["choice"] # 'billing'
r["answers"]["urgent"]["noul"] # P(yes)
r["answers"]["severity"]["score"] # expected level 0..2
m.predict_batch(states, questions) # many texts, same questions
m.learn("team", questions["team"], [(text, "billing"), ...]) # few-shot memory, no training
Every answer carries probabilities, confidence and answer_confidence (the calibrated probability of the
reported answer); pass min_confidence= to get an abstained flag. Also included:
python -m nirnay.server — HTTP server with POST /v1/systemone and /v1/systemone/batch (Jev / Laya wire format,
so existing clients only change their base URL)python -m nirnay.mcp_server — MCP tools decide, classify, yes_no for local agentsnirnay — command linestate (text / JSON) question "<type> question: <instructions>" options "<label>: <description>"
| | |
encoder layers 1-8 (frozen) — every text encoded on its own; question/option results cached per schema
| | |
encoder layers 9-12 (trained copies) over [ state ; questions ; options ] with an asymmetric mask:
state rows attend to the state only -> computed ONCE per call, shared by every question
question rows attend to state + own question
option rows attend to state + their question + own option (never to other options)
|
score at each option's first token + learned cosine prior -> softmax over options -> per-bucket temperature
The top of the network is a cross-encoder (option tokens read the state at every layer), but the state never reads
the questions — so one state encoding serves every question, and options are independent of each other.
Training minimises strictly proper scoring rules (log score, plus ranked probability score for ratings) with exact
gradients. Layers 1-8 are frozen, so only the top 4 layers + scorer train (29M parameters): v3.1 continued from v2
for 1.25 epochs on 324k typed questions, on a CPU (two sockets, data-parallel, ~9 hours). The repository also ships
a GPU trainer (code/train/train_gpu.py, verified to reproduce the CPU pipeline) and a free-Kaggle-GPU job.
Laya (laya 0.4.14 from PyPI, all three checkpoints, its own API and calibration) ran on the same machine on
byte-identical cases: the 400-case suites of Laya's own application benchmark (seed 13) plus two held-out extras.
Jev is a closed cloud API, so its column is the published third-party figure.
| task | Nirnay v3.1 | Nirnay v2 | Nirnay v1 | Laya en | Laya ml | Laya td | Jev (published) |
|---|---|---|---|---|---|---|---|
| AG News (4 topics) | 0.925 | 0.925 | 0.882 | 0.950 | 0.935 | 0.953 | 0.910 |
| DAIR emotion (6) | 0.595 | 0.550 | 0.515 | 0.593 | 0.537 | 0.600 | 0.480 |
| Banking77 (77 labels offered) | 0.598 | 0.610 | 0.595 | 0.425 | 0.470 | 0.492 | 0.870 (72 labels, n=100) |
| support triage (10 queues) | 0.407 | 0.415 | 0.417 | 0.502 | 0.540 | 0.505 | — |
| Enron spam | 0.995 | 0.995 | 0.968 | 0.998 | 0.998 | 0.968 | — |
| phishing | 0.973 | 0.975 | 0.948 | 0.980 | 0.993 | 0.940 | — |
| jailbreak (toxic-chat) | 0.647 | 0.695 | 0.787 | 0.708 | 0.805 | 0.762 | — |
| toxicity (toxic-chat) | 0.688 | 0.660 | 0.595 | 0.530 | 0.535 | 0.530 | — |
| RAG passage relevance | 0.680 | 0.655 | 0.613 | 0.625 | 0.672 | 0.625 | — |
| model routing (domain) | 0.875 | 0.404 | 0.446 | 0.639 | 0.441 | 0.659 | — |
| SST-5 (5-level rating) | 0.490 | 0.492 | 0.425 | 0.352 | 0.292 | 0.470 | — |
| SMS spam | 0.790 | 0.670 | 0.600 | 0.938 | 0.807 | 0.910 | 0.930 (independent) |
Banking77's cases are the first 400 test messages (as in Laya's benchmark), which cover 10 of the 77 intents; all 77 are offered in every question. The routing cases cover 3 of its 6 domains.
Honest summary: against the best of Laya's three checkpoints (picked per task, after the fact) Nirnay v3.1 wins 5 tasks (Banking77, toxicity, RAG relevance, model routing, SST-5), ties 2 (emotion, Enron spam) and loses 5 (AG News, support triage, phishing, jailbreak, SMS spam); against Laya's default English checkpoint it is also 5 / 2 / 5. It beats Jev's published AG News and emotion figures. Laya is a 421M-parameter model fully fine-tuned on GPUs; Nirnay trains 29M parameters on a CPU.
Yes / no ranking quality (AUC, threshold-free; v3.1 vs v2): jailbreak 0.701 vs 0.713 · toxicity 0.787 vs 0.765 · SMS spam 0.919 vs 0.899 · RAG relevance 0.744 vs 0.727 · phishing 0.998 vs 0.997 · Enron spam 1.000 vs 1.000.
| speed (warm p50) | 1 question | 10 questions | 50 questions |
|---|---|---|---|
| Nirnay v3.1 | 129 ms | 232 ms | 655 ms |
| Nirnay v1 (fast) | 54 ms | 95 ms | 258 ms |
| Laya at its fastest (any checkpoint, eager or its ONNX backend) | 159 ms | 1,389 ms | 6,730 ms |
Does Laya's order sensitivity explain the Banking77 gap? The same 400 cases, with the 77 options in the suite's own (alphabetical) order and in 5 random orders, each order used for every case:
| Banking77, 6 option orders | accuracy by order | same answer in all 6 orders | probabilities averaged over the 6 orders | right in at least one order |
|---|---|---|---|---|
| Nirnay v2 | 0.610 in every order | 100% | 0.610 | 0.610 |
| Nirnay v3.1 | 0.598 in every order | 100% | 0.598 | 0.598 |
| Laya td (its best checkpoint on this task) | 0.435 to 0.603 (mean 0.493; own order 0.492) | 26.8% | 0.552 | 0.695 |
In its luckiest order Laya nearly matches Nirnay, and averaging its probabilities over the six orders (6x the compute) closes about half the gap; choosing the lucky order would need the labels. Each of the 10 intents is 10% of the score: in one order Laya's accuracy on "exchange rate" alone fell from 0.975 to 0.200. Script and per-case outputs: bench/order_check.py and bench/results/order_check/.
few-shot learn(), Banking77, examples per intent |
0 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| Nirnay v3.1 | 0.598 | 0.635 | 0.725 | 0.825 | 0.840 |
| Nirnay v2 | 0.610 | 0.683 | 0.768 | 0.840 | 0.853 |
Determinism: repeat calls byte-identical; across 1/4/8/16 CPU threads no decision changed (probabilities differ by at most 0.0001). Cold start to first answer ~7 s (Laya 12–16 s). Full tables (calibration, footprint, ablations, v3 → v3.1): REPORT.md.
learn().learn()) on your own traffic before automated gating.Public datasets listed in the metadata, converted to typed questions with varied wording, label subsets, yes/no
reformulations and ordinal ratings (code/train/build_data.py). Held out of training, mirroring Laya's own policy:
dair-ai/emotion, mteb/banking77, lmsys/toxic-chat, gsm8k / mbpp (the routing benchmark's sources), SetFit/sst5,
ucirvine/sms_spam (and any text matching it), XNLI, MASSIVE, SIB-200 and all jailbreak / prompt-injection datasets;
any text appearing in a benchmark case was removed by exact match.
New in v3.1: CodeAlpaca (CC-BY-4.0), HumanEval (MIT), SVAMP (MIT), AQuA-RAT (Apache-2.0), MATH (MIT), sql-create-context (CC-BY-4.0), Gretel synthetic text-to-SQL (Apache-2.0), WritingPrompts (MIT), Dolly 15k (CC-BY-SA-3.0), NQ-Open (CC-BY-SA-3.0), SODA (CC-BY-4.0), India SMS spam (MIT), ham-spam-scam (MIT, small sample), NVIDIA Aegis 2.0 (CC-BY-4.0). Some training datasets carry research-only or non-commercial terms (for example MS MARCO, RACE, Yahoo Answers, Yelp reviews, Quora question pairs). The code and weights are released under Apache-2.0; check those datasets' terms before commercial use.
nirnay_config.json, encoder.safetensors, head.safetensors, tokenizer.json — v3.1 (default)v2/ — the previous default · fast/ — v1 (bge-small)dist/nirnay-1.0.0-py3-none-any.whl — the inference packagecode/ — full source: package, data builder, CPU and GPU training, Kaggle job, benchmark harness, testsREPORT.md — every measurementBuilt on BAAI/bge-base-en-v1.5 and BAAI/bge-small-en-v1.5 (MIT). The benchmark cases reproduce the application benchmark of Laya (source) by Nandakishor M (Convai Innovations), whose open model inspired this project.
encoder.safetensors 436 MBv2/encoder.safetensors 436 MBfast/encoder.safetensors 133 MBhead.safetensors 116 MBv2/head.safetensors 116 MBfast/head.safetensors 29 MBBAAI/bge-base-en-v1.5