N-Q-26M-JEV-1027

The 26M decision model, October 27 update: much better at logic, false premises and grounding, plus routing decisions for chat

[!NOTE] Versioning. Like DeepSeek's V3-0324 or R1-0528, the suffix is the release date, MMDD: 1027 = October 27, 2026. Same model family and interface as N-Q-26M-JEV, new weights. Later updates get their own date; nothing is overwritten.

[!IMPORTANT] A research release, not a product. It is a 26M model: strong on the decision families it was trained on, far behind billion-parameter decision models on general reasoning (see Honest results).

What changed since N-Q-26M-JEV

Benchmarks (held out: none of these were trained on)

Same items and protocol for every model in the table; the small cross-encoders were run by us (eval_bench_xenc.py), the openjev-4B row is from its model card.

What this says, plainly:

• Adversarial NLI and false premises: the 26M model now beats 3× larger NLI cross-encoders (ANLI r1 43 vs 31–36, FalseQA 0.61 vs 0.52) and is level with the 150M ModernCE on FalseQA.
• Plain NLI improved by 8 points but is still the biggest gap to the specialised models (73 vs 86–90): they were trained on ~1M NLI pairs for several epochs, this model on ~0.3M.
• JevBench: better on the hard tier (+2.7), worse on the standard tier (−4.2), the same overall. At this size every model we measured lands around 46–52%; the 4B models are in a different league.
• LLMBar (which of two answers follows the instruction) stays near chance (50%) for all small models.

Trained decision families

Held-out test splits, symmetric mode. ECE = expected calibration error (0 = confidence matches accuracy).

Mean ECE over the families: 0.065 (the previous release: 0.05).

Solution judge. "Is this solution's final answer right?" separates right from wrong solutions of the N-Q-26M chat model with AUROC 0.98, but most of that comes from telling easy problems from hard ones: picking the best of 8 samples of the same problem lifts accuracy only from 14.5% to 16.0% (the ceiling, "any of 8 is right", is 22.3%). A pairwise "which of two solutions is right" head was also trained, on too few pairs (825), and does not help yet.

Usage

from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-26M-JEV-1027", trust_remote_code=True)
jev = AutoModel.from_pretrained("q-project/N-Q-26M-JEV-1027", trust_remote_code=True).eval()

jev.decide(tok, "Write a Python function that returns the n-th Fibonacci number.", {
    "topic": {"kind": "choice", "question": "What is this request about? Pick the topic.",
              "options": ["coding", "math", "writing", "cooking", "travel", "health"]},
    "level": {"kind": "score", "question": "How difficult is this request?", "tiers": ["easy", "medium", "hard"]},
})
# topic: coding 0.956 · level: medium 0.843

# chat routing (new)
q = "Понадобится ли для ответа точное вычисление калькулятором?"
jev.noul(tok, "Сколько будет 347 умножить на 86?", q, labels=("нет", "да"))            # 0.99999
jev.noul(tok, "Расскажи, почему небо голубое.", q, labels=("нет", "да"))               # 0.00007
q = "Нужно ли рассуждать перед ответом на этот запрос?"
jev.noul(tok, "У Пети 3 яблока, у Маши в 2 раза больше. Сколько у них вместе?", q, labels=("нет", "да"))  # 0.967
jev.noul(tok, "Привет! Как дела?", q, labels=("нет", "да"))                            # 0.002

# grounding: is the claim supported by the text?
jev.noul(tok, "Text: The Eiffel Tower was completed in 1889 and is 330 metres tall.\nClaim: The Eiffel Tower is over 300 metres tall.",
         "Does the text support the claim?")      # 0.81
jev.noul(tok, "Text: The Eiffel Tower was completed in 1889 and is 330 metres tall.\nClaim: The Eiffel Tower was completed in 1925.",
         "Does the text support the claim?")      # 0.03

# NLI
jev.choice(tok, "Premise: A man is playing a guitar on stage.\nHypothesis: Nobody is making music.",
           "How does the hypothesis relate to the premise?", ["entailment", "neutral", "contradiction"])   # contradiction 0.993

• The phrasings above are the ones the model was trained with; other wordings work, these work best.
• symmetric=True (default) also scores the options in reverse order and averages (the backbone is causal).
• States up to 1536 tokens are read in full; longer ones keep their beginning (NQ4ForDecision.encode(..., max_state=1536)).
• Input layout, if you call the head directly: <bos><USER>state<EOT><R12>question<R11>option_1<R11>…option_K<R11>.

Training

Continued from N-Q-26M-JEV on one V100, full fine-tune with the decision head.

Loss: cross-entropy to 0.5 · one-hot + 0.5 · teacher where a teacher distribution exists, plain log-loss elsewhere.

Packed single-file formats

model.qpack (QPACK1, 89 MiB) and model.q8.qpack (QPACK2, int8, 46.5 MiB) load through AutoModel.from_pretrained(..., trust_remote_code=True) from a folder without *.safetensors, or with from_qpack(...); both pass python qpack.py --selftest (5/5), and both run in the native qpack C++ engine and the browser runtime.

Limitations

• 26M parameters: no world knowledge, no multi-step reasoning; task types far from the training families are close to chance.
• Plain NLI (73%) is well below specialised 70–150M NLI models; the routing and grounding heads are the strong part.
• English for the openjev families (that data is English); Russian for routing, toxicity, sentiment; both for NLI.
• Answer-quality and Russian sentiment labels are noisy, and so are the model's judgements there.
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check them before commercial use. In particular openjev-data is built from the train splits of SNLI, MultiNLI, ANLI (CC BY-NC 4.0), WANLI, QNLI, NLI-FEVER, RAGTruth, MiniCheck, FalseQA, xLAM and others; plus saiga_scored, saiga_preferences, UltraFeedback, SmolTalk, GrandMaster-PRO-MAX, XNLI, the RU sentiment collection, SST-2, ru_paradetox_toxicity, AG News, FalseQA, and the pretraining data of N-Q-26M. Teacher: dleemiller/ModernCE-base-nli.