solvi-ai/solvi-large (preview)
A 396M-parameter ModernBERT-large cross-encoder that answers typed questions about a text or a JSON state for
solvi: one option, several options, ordered scores, yes / no, not stated, spans,
evidence quotes, rankings and numbers (bins), with a calibrated confidence and an act / escalate signal per answer.
Same format as the 150M solvi-base (solvi_decide v2, subformat l14g typed v2); this is the larger, stronger variant.
Honest summary. It is clearly better than our previous deciders on every test, and better than GLiNER2.5-Decide on typed questions over JSON states — but it does not beat GLiNER2.5-Decide on zero-shot choice questions (Fast Decisions dev 59.4% vs 62.9%). With 64 labelled examples per head it reaches 63.0%. It is also slower: 137 ms per question on a CPU (ONNX fp16).
Results (held-out tests; never used for training or checkpoint selection)
Criteria were registered before training (research log L19; 4 of 10 met for this model — the misses are under Limitations). The Fast Decisions numbers are on the public dev split (100 rows × 17 domains) in our harness; Fastino's official scores (GLiNER2.5-Decide 60.1, …) are on a hidden test split and are not directly comparable.
Choice questions — Fast Decisions (dev, macro over domains):
| model | params | zero-shot | + bias correction, unlabelled (Zc) | + 32 labelled (Sc) | + 64 labelled (Sc) |
|---|---|---|---|---|---|
| GLiNER2.5-Decide (fastino) | 340M | 62.9% | — | — | — |
| solvi-large (this model) | 396M | 59.4% | 60.5% | 62.0% | 63.0% |
| solvi-base (distilled student) | 150M | 56.3% | 58.2% | 60.2% | 60.4% |
| L14g (previous base, not published) | 150M | 56.2% | 57.4% | 58.8% | — |
| solvi L14d (previous clean decider) | 150M | 57.3% | 59.5% | 61.4% | 61.9% |
| knowledgator/gliformer-large-v1 (our run) | 576M | 43.9% | — | — | — |
| Laya (our earlier run) | — | 48.9% | — | — | — |
Single-label heads alone: 63.4%; exact multi-label sets: 26.3%.
Typed questions over JSON states — typed-decisions test (2000 questions):
| model | zero-shot | fitted per process, 32 / 300 examples |
|---|---|---|
| solvi-large | 54.5% | 63.1% / 64.9% |
| GLiNER2.5-Decide (our run) | 50.3% | — |
| L14g (previous base) | 45.1% | 58.8% / 61.3% |
| Laya base | 36.0% | fine-tuned on this dataset: 76.7% |
typed-decisions gold comes from a ~4B teacher's judgement, so "accuracy" here means agreement with that teacher.
New kinds on real text:
| test | measure | result |
|---|---|---|
| ContractNLI test (in-distribution: ContractNLI train was used for training) | yes / no / not stated | 88.8% |
| ContractNLI | the evidence quote supports the answer (entailed / contradicted pairs) | 62.0% (previous base L14g: 23%) |
| Taskmaster-2, held-out slots | span exact / token F1 | 70.2% / 0.83 |
| Taskmaster-2 | "not stated" (noisy gold) | 35.5% |
Synthetic states with 9 held-out schemas: all kinds 97.9% (prose rendering 98.6%); yes / no / not stated 98.9%; span exact 99.4%; evidence quotes literally correct 100% and supporting 99.5%; ranking NDCG@3 0.983; number median bin 95.8%.
Act / escalate signal (AUROC of "the answer is right"): ContractNLI 0.85, Fast Decisions 0.76, Taskmaster-2 0.74,
typed-decisions 0.67, synthetic 0.93. The shipped act calibrator is fitted on development data and is over-confident on new
real text — set the threshold on your data with act_guard (below).
Calibration (ECE): ContractNLI 0.026, Taskmaster-2 0.062, synthetic ≤ 0.02 per kind (rank 0.21), typed-decisions 0.21, Fast Decisions 0.21 (fit on your task before trusting zero-shot confidences).
Speed (CPU, 4 threads, idle machine): ONNX fp16, one question with 10 options and ~20 words: 137 ms (fp32 128 ms); a question over a long JSON state: 450–740 ms. ONNX = PyTorch on 300 of 300 Fast Decisions heads.
Several questions per pass (block layout) — answers agree with one-question-per-pass only 90.8% (typed-decisions) /
90.2% (Fast Decisions) / 97.5% (synthetic) of the time, so solvi_decide.json ships with multi_question.enabled = false.
The block ONNX export (onnx/model_block_fp16.onnx) is included for experiments (same answer as PyTorch 99.3%).
Training
- Initial weights: answerdotai/ModernBERT-large (Apache-2.0).
- Recipe: 200 minutes on one GPU per run (≈ 9.7k updates, effective batch 128, layer-wise LR decay 0.92), a first stage with 75% classification data, then the full mix; 35% of batches in the block layout with a block/full consistency loss. This model is the weight average of the best and the last checkpoint of that run, chosen by a development metric (8 permissive held-out classification sets + development splits of the training sources) — never by the tests above. Three sibling runs (a capacity-aware single-stage mix, a run without GLiNER distillation, a clean-data annealing of the second run) scored lower on that metric; averaging across runs made it worse.
- Teachers (labels only on the license-clean texts below; never on test data), all Apache-2.0: Qwen/Qwen2.5-32B-Instruct, Qwen/Qwen2.5-14B-Instruct, mistralai/Mistral-Small-24B-Instruct-2501, fastino/GLiNER2.5-Decide. Soft targets = vote shares; questions without ≥ 2 agreeing votes were dropped. On heads with real gold the consensus was right 87.5% of the time (GLiNER alone 73.3%).
- Data agent: Qwen2.5-32B-Instruct generated 22.7k texts with classification schemas over 76 business / content domains × 28 task types × 26 layouts; each head was verified by Qwen2.5-14B and Mistral-Small (kept at ≥ 2 of 3). The domain matrix was written from scratch and overlaps the public category descriptions of Fast Decisions; no Fast Decisions row was used, and all generated texts were de-duplicated against the Fast Decisions dev split (0 near-duplicates).
- De-duplication: exact + MinHash (Jaccard ≥ 0.5) against Fast Decisions dev, typed-decisions test, ContractNLI test and Taskmaster-2 test texts; 216 ContractNLI-train windows too close to test contracts were removed.
Training data and attribution
The full manifest with licenses, URLs and counts is LICENSES.json. Policy: permissive, public-domain and share-alike data (CC BY-SA,
with attribution here); excluded: non-commercial, research-only, unlicensed, or terms that forbid training.
| source | license | attribution |
|---|---|---|
| solvi synthetic corpora (operational texts, typed states, judgment states; own generators, no external text) | Apache-2.0 | solvi authors |
| SQuAD 2.0 (train) | CC BY-SA 4.0 | Pranav Rajpurkar, Robin Jia, Percy Liang: "Know What You Don't Know: Unanswerable Questions for SQuAD", ACL 2018 |
| BoolQ (train) | CC BY-SA 3.0 | Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova: "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions", NAACL 2019 |
| CFPB Consumer Complaint Database | CC0-1.0 | Consumer Financial Protection Bureau |
| English Wikinews (2023-07-28) | CC BY 2.5 | Wikinews contributors, https://en.wikinews.org |
| arXiv abstracts (metadata) | CC0-1.0 | arXiv.org |
| Taskmaster-1 | CC BY 4.0 | Google Research |
| MultiWOZ 2.2 | MIT | Budzianowski et al. |
| Amazon polarity | Apache-2.0 | Zhang et al.; McAuley & Leskovec |
| GoEmotions | Apache-2.0 | Google Research |
| LEDGAR, UNFAIR-ToS (LexGLUE) | CC BY 4.0 | Tuggener et al.; Lippi et al.; Chalkidis et al. |
| SMS Spam Collection | CC BY 4.0 | Almeida & Hidalgo (UCI) |
| Toxic conversations 50k (Civil Comments / Jigsaw) | CC BY 4.0 | Jigsaw / Conversation AI; MTEB |
| deepset prompt-injections | Apache-2.0 | deepset |
| CLINC150 | CC BY 3.0 | Larson et al. |
| Banking77 | CC BY 4.0 | PolyAI |
| MASSIVE (en-US) | CC BY 4.0 | Amazon Science |
| SNIPS NLU | Apache-2.0 | Snips |
| Bitext customer-support, retail-banking, travel, insurance | CDLA-Sharing-1.0 | Bitext Innovations |
| VitaminC (includes FEVER-based claims) | CC BY-SA 3.0 | Tal Schuster, Adam Fisch, Regina Barzilay: "Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence", NAACL 2021; Wikipedia contributors | | ContractNLI (train split only) | CC BY 4.0 | Yuta Koreeda, Christopher D. Manning, "ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts", Findings of EMNLP 2021; Hitachi America, Ltd. | | LLM-generated texts and labels (Qwen2.5-32B / 14B-Instruct, Mistral-Small-24B-Instruct-2501) | Apache-2.0 (model licenses) | Qwen team (Alibaba Cloud); Mistral AI |
SQuAD 2.0, BoolQ and VitaminC are share-alike datasets: this card is their attribution. The model weights are released under Apache-2.0; if you redistribute the datasets themselves, their own licenses apply.
Checkpoint selection only (not trained on): DBpedia-14 (CC BY-SA 3.0), HuffPost News Category (CC BY 4.0), zeroshot twitter-financial-news topic / sentiment (MIT), Civil Comments (CC0), poem_sentiment (CC BY 4.0), LexGLUE SCOTUS (CC BY 4.0), Amazon counterfactual (CC BY 4.0).
Independent benchmarks (as released, zero-shot)
| benchmark | solvi-large | solvi-base | others (their published numbers or the benchmark's leaderboard) |
|---|---|---|---|
| jabr classifier-benchmark v2 (49 tasks, 866 cases), macro | 0.694 | 0.602 | Jev 0.966, GLiNER2.5-Decide 0.739, Von 0.720, GLiNER2 0.684, Laya 0.583 |
| decision-models-under-pressure: accuracy with 128 options | 45.1% | 50.8% | Jev 60.0%, best other open model 41.0%, Laya 38.5% |
| same: answers changed by reordering 64 options | 40.9% as listed · 0.5% in solvi ≥ 0.5.1 (sorted order by default) | 24.1% as listed | Jev 14.6%, Laya 49.4% |
| same: accuracy lost to near-duplicate distractors (64 options) | 0.260 | 0.247 | Jev 0.105, Laya 0.345 |
Yes / no questions are the weak spot on jabr: the model ranks them reasonably but says "yes" 36% of the time where the gold
labels have 50% — calibrate the threshold on your data (act_guard, fit).
Escalation with a guarantee (solvi ≥ 0.5.1)
The act threshold shipped in solvi_decide.json was fitted on development data and does not keep its promise on new real
text. Calibrate on a few hundred labelled examples of your own stream instead:
part.act_guard(examples, risk=0.10) # P(answered alone and wrong) ≤ 10% of all questions, for inputs like the examples
Measured with 300 calibration examples per data set and 200 random splits (the rest of the set is the test). "Answered" is the share decided without a person, "error" the error among those, "risk" the share of all questions answered alone and wrong — the number the guarantee is about:
| data set | shipped "10%" threshold: answered / error | act_guard(risk=0.10): answered / error / risk |
act_guard(risk=0.05) |
|---|---|---|---|
| typed-decisions | 78% / 39% | 32% / 30% / 9.8% | 18% / 27% / 5.0% |
| Taskmaster-2 | 81% / 32% | 50% / 20% / 10.0% | 34% / 14% / 4.9% |
| ContractNLI | 98% / 10% | 97% / 10% / 9.6% | 85% / 6% / 5.1% |
| JSON, 9 held-out schemas | 100% / 2% | 99.6% / 1.8% / 1.8% | 99.7% / 1.9% / 1.9% |
The guarantee holds on every set; how much can be automated depends on how hard the questions are. It holds for inputs like the calibration examples, not under a shift of domain: recalibrate when your inputs change.
Evaluation only — never trained on: Fast Decisions (fastino, dev split), typed-decisions test, ContractNLI test, Taskmaster-2 held-out domains.
Limitations
- Not better than GLiNER2.5-Decide on zero-shot choice questions (59.4% vs 62.9% on Fast Decisions dev). It reaches GLiNER's zero-shot level only after fitting on ~64 labelled examples per head.
- Slower than 100 ms on a CPU (137 ms per short question, ONNX fp16); use solvi-base (50 ms) where latency matters.
- The act / escalate calibrator is over-confident on new real text (at the "≤ 10% error" threshold fitted on development data the
real error was 32–39% on Fast Decisions, typed-decisions and Taskmaster-2; 10% on ContractNLI). Use
act_guardon your data (solvi ≥ 0.5.1): the risk it promises held on every set we measured. - Several questions per pass disagree with one-question-per-pass in ~9% of answers on real states — disabled by default.
- ContractNLI numbers are in-distribution (train split used); evidence on other document types is weaker.
- "Not stated" in dialogues is under-recognised (35.5% on Taskmaster-2); rank confidences are poorly calibrated (ECE 0.21).
- English only; inputs longer than 512 tokens (1024 in the block layout) are truncated — only the text, never the question.
- Not for medical, legal or credit decisions on its own; in solvi those rules belong in hard checks with escalation to a person.
Files
model.safetensors (bf16), onnx/model_fp16.onnx (full layout: input_ids, attention_mask → logits [B, L, 6]),
onnx/model_block_fp16.onnx (block layout: input_ids, position_ids, full_attention_mask, sliding_attention_mask),
onnx/parity.json, config.json, tokenizer.json, tokenizer_config.json, solvi_decide.json (capabilities, temperatures,
act calibrator, multi_question.enabled = false), l14g_format.py and l14f_format.py (the exact input / output format),
LICENSES.json (training data manifest).
Citation
@software{solvi,
title = {solvi: verifiable decision systems from catalogs of functions and checks},
author = {mxkuzn and solvi contributors},
year = {2026},
url = {https://github.com/solvi-ai/solvi}
}