Ask choice, yes / no and rating questions about text in any of ~100 languages and get calibrated probabilities in
one pass — no GPU, no text generation, order-proof, deterministic. Same API and question format as
Nirnay and Laya / Jev (/v1/systemone). Write the question in English;
the text can be in any language.
| held-out benchmark (zero-shot) | Nirnay multilingual | Laya multilingual | Nirnay v3.1 (English) |
|---|---|---|---|
| MASSIVE intents: 51 languages, all 60 intents as options | 0.386 | 0.217 | 0.076 |
| per language vs Laya multilingual | 49 wins, 1 tie, 1 loss | ||
| SIB-200 topics: 205 languages and scripts, 7 topics | 0.670 | 0.632 | 0.411 |
| per language vs Laya multilingual | 105 wins, 32 ties, 68 losses | ||
| option-order flip rate on these suites | 0.000 | 0.185 | 0.000 |
| CPU time per case on these suites | 89 ms | 138 ms |
Laya multilingual (laya 0.4.14, its mmBERT-base checkpoint, its own API) ran on the same machine on byte-identical
cases. Neither model trained on MASSIVE or SIB-200. MASSIVE asks one 60-way question per request with no examples —
hard for any zero-shot model (Nirnay's absolute score is modest; Laya's fixed option budget struggles more). Examples
from MASSIVE: Hindi 0.350 vs 0.175, Spanish 0.400 vs 0.175, Malayalam 0.463 vs 0.087, Tamil 0.412 vs 0.125,
Urdu 0.438 vs 0.100, Bengali 0.450 vs 0.163; Chinese (simplified) is the one loss, 0.338 vs 0.412.
nandukmelath/nirnay (v3.1) — more accurate
in English on most tasks, faster and half the size.pip install git+https://github.com/nandukmelath/nirnay
import nirnay
m = nirnay.load("nandukmelath/nirnay-multilingual")
q = {"intent": {"type": "choice", "instructions": "What does the user want?",
"criteria": ["set an alarm", "check the weather", "play music", "order food", "book a taxi"]},
"urgent": {"type": "noul", "instructions": "Is this request urgent?"}}
m.predict({"message": "¿Me puedes despertar mañana a las seis y media?"}, q)["answers"]["intent"]["choice"]
# 'set an alarm' (confidence 1.00)
m.predict({"message": "എനിക്ക് ഇപ്പോൾ തന്നെ എയർപോർട്ടിലേക്ക് ഒരു ടാക്സി വേണം"}, q)["answers"]
# intent 'book a taxi' (0.98), urgent P(yes) 0.65
Real outputs of this model: Spanish → set an alarm, Hindi "कल दिल्ली में बारिश होगी क्या?" → check the weather,
Malayalam → book a taxi, German "Spiel bitte etwas ruhige Jazzmusik" → play music, Arabic "أريد أن أطلب بيتزا كبيرة"
→ order food. predict_batch, learn() (few-shot memory), the HTTP server and the MCP server work exactly as in
Nirnay.
| task | multilingual | Nirnay v3.1 | best Laya checkpoint |
|---|---|---|---|
| AG News (4 topics) | 0.943 | 0.925 | 0.953 |
| DAIR emotion (6) | 0.485 | 0.595 | 0.600 |
| Banking77 (77 labels offered) | 0.477 | 0.598 | 0.492 |
| support triage (10 queues) | 0.425 | 0.407 | 0.540 |
| Enron spam | 0.995 | 0.995 | 0.998 |
| phishing | 0.975 | 0.973 | 0.993 |
| jailbreak (toxic-chat) | 0.755 | 0.647 | 0.805 |
| toxicity (toxic-chat) | 0.657 | 0.688 | 0.535 |
| RAG passage relevance | 0.660 | 0.680 | 0.672 |
| model routing (domain) | 0.857 | 0.875 | 0.659 |
| SST-5 (5-level rating) | 0.480 | 0.490 | 0.470 |
| SMS spam | 0.868 | 0.790 | 0.938 |
Against the best of Laya's three checkpoints in English: 3 wins (toxicity, model routing, SST-5), 1 tie, 8 losses. Banking77's cases are the first 400 test messages (as in Laya's benchmark), which cover 10 of the 77 intents; all 77 are offered in every question.
| speed (warm p50, same CPU) | 1 question | 10 questions | 50 questions |
|---|---|---|---|
| Nirnay multilingual | 146 ms | 297 ms | 829 ms |
| Nirnay v3.1 (English) | 129 ms | 232 ms | 655 ms |
| Laya at its fastest (any checkpoint, eager or ONNX) | 159 ms | 1,389 ms | 6,730 ms |
Determinism: repeat calls byte-identical; across 1/4/8/16 CPU threads no decision changed (probabilities differ by at
most 0.0001). Few-shot learn(), Banking77: 0.478 with no examples → 0.768 with 8 per intent (0.745 with 16).
Full tables: REPORT.md.
The same architecture as Nirnay (deep fusion: the encoder's top 4 layers run over [state ; questions ; options] with an asymmetric mask, so the text is read once per call and options never see each other), on intfloat/multilingual-e5-base (XLM-RoBERTa, 100 languages). Trained on a CPU with proper scoring rules: ~1.5 epochs over 385k typed questions on both sockets of a 2014 workstation.
learn().Everything in Nirnay v3.1's mix (see its card), plus: MTOP assistant intents in 6 languages (tasksource/mtop, CC-BY-SA-4.0 — a different dataset from MASSIVE, with similar assistant intents), PAWS-X paraphrases in 6 languages, machine-translated MNLI in 26 languages (MoritzLaurer/multilingual-NLI-26lang-2mil7), TextDetox toxicity in 14 languages (OpenRAIL++), language identification in 20 languages (papluca), multilingual Amazon reviews and multilingual tweet sentiment. Held out: MASSIVE, SIB-200, XNLI, and everything v3.1 holds out (emotion, Banking77, toxic-chat, gsm8k, mbpp, SST-5, the UCI SMS collection, all jailbreak datasets); any text appearing in a benchmark case was removed.
Some training datasets carry research-only or non-commercial terms (for example the multilingual Amazon reviews, MS MARCO, RACE, Yahoo Answers, Yelp reviews). The code and weights are released under Apache-2.0; check those datasets' terms before commercial use.
Built on intfloat/multilingual-e5-base (MIT). The benchmark cases reproduce the application benchmark of Laya by Nandakishor M (Convai Innovations), whose open model inspired this project.
encoder.safetensors 1.1 GBhead.safetensors 116 MBintfloat/multilingual-e5-base