decosa-number-checker-modernbert-large (v2)
Given a claim sentence and its evidence (a table, a passage or both), it says whether the claim's numbers are supported, contradicted or not verifiable. For a contradiction it names the error type (wrong value, unit or scale, period, direction, arithmetic, or entity) and points at the evidence cell it read.
It is the learned tier of a deterministic number check. The code in numparse.py extracts every figure in the claim
(amounts with scale words, percentages, percentage points, fractions such as "about a fifth", multiples such as
"nearly doubled") and searches the evidence for a derivation: a cell, a change, a percentage change, a share, a sum,
a point change. The model reads the claim, those findings and the evidence, and decides what the code cannot: whether
the derivation is about the right line, period and direction, and what an untraced figure means.
- Weights: Apache-2.0. Base: ModernBERT-large (Apache-2.0).
- Size: 395M parameters;
model.safetensors1.58 GB; runs on a CPU (median 453 ms a claim on 4 threads; v1 on ModernBERT-base took 135 ms). - sha256 of every file:
SHA256SUMS; the evaluation numbers below are also ineval_summary.json. - Versions: v2 is the first public release. v1 was an earlier internal model on ModernBERT-base; it is shown in the tables for reference and is not released.
- It needs this repository's
modeling_checker.pyandnumparse.py(custom heads and the code tier); it is not a stocktransformerspipeline.
Use
# pip install torch transformers safetensors
from modeling_checker import NumberChecker # this repository's modeling_checker.py and numparse.py
nc = NumberChecker.from_dir(".")
nc.check("Revenue rose by about a fifth in 2020.",
table=[["", "2019", "2020"], ["Revenue", "1,000", "1,210"]], unit="millions")
# {'verdict': 'supported', 'p': {...}, 'code': 'traced', 'error_type': None, 'pointer': {...}}
nc.check("Revenue fell 21% in 2020.", table=[["", "2019", "2020"], ["Revenue", "1,000", "1,210"]])
# {'verdict': 'contradicted', 'code': 'mismatch', 'error_type': 'direction', ...}
Rules we use in production, and recommend:
- A hard code mismatch is final (
code == "mismatch": a thousand-fold slip against a declared unit, the other period's figure for the same line, a signed change stated in the wrong direction). The model never overrules it. - When the code traced every figure, flag the claim only when the model is very sure (0.813 or more on "contradicted").
- Otherwise take the model's answer at 0.78 or more, and send the rest to a person or a larger model.
onnx/model.onnx is the same network exported for ONNX Runtime (inputs input_ids, attention_mask; outputs
verdict, error, pointer logits). Divide the verdict logits by temperature in checker.json before the softmax.
Training data
About 80,000 claim/evidence pairs, built by us. No non-commercial data was used at any stage, and nothing from QuanTemp (CC BY-NC).
| Source | Licence | What we used |
|---|---|---|
| SEC Financial Statement Data Sets 2026q1 (EDGAR XBRL face statements, 4,489 filings) | US government work, public domain (17 U.S.C. 105) | statement tables; claims written and planted by our generator |
| FinQA (github.com/czyssrs/FinQA) | MIT | tables and passages from its train split; claims by our generator |
| TAT-QA (github.com/NExTplusplus/TAT-QA) | CC BY 4.0 (data), MIT (code) | tables and passages from its train split; claims by our generator |
| TabFact (github.com/wenhuchen/Table-Fact-Checking) | MIT (dataset); tables from Wikipedia, CC BY-SA | 10,000 human claims with a digit, train split |
| ClinicalTrials.gov results (API v2, 439 phase 3 interventional trials with results, retrieved 27 Sep 2026) | US government, public domain; attribution and a statement of changes below | results and adverse-event tables; lay-language claims by our generator |
| Decosa synthetic SAR case files | ours (generated from a seed by code; fictional people, accounts and banks) | ledger rows and narrative sentences, with the numeric block's own findings |
FinQA's tables come from FinTabNet (IBM, CDLA-Permissive-1.0), which is built from S&P 500 companies' annual reports; TAT-QA's tables and passages also come from public financial reports; TabFact's claims were written by crowd workers. No training text was written by a closed-API model; none of the training data ships with this repository.
ClinicalTrials.gov. Data retrieved from ClinicalTrials.gov (National Library of Medicine), API v2, on 27 September 2026. Changes: we parsed the results and adverse-event tables, cut them to the rows a claim concerns, and generated claim sentences about them, some with planted numeric errors. The results were not reviewed by NLM, and ClinicalTrials.gov does not endorse this model.
Contradictions were planted in code with known labels: a value changed by 3-30%, a thousand-fold scale slip, the other period's figure, a flipped rise/fall, rounding beyond the stated precision, percent written for percentage points, a percentage on the wrong base, a sum missing a term, another line's or group's figure. Not-verifiable pairs remove the claim's rows from the evidence. TabFact adds human-written claims (entailed or refuted) over Wikipedia tables.
Evaluation
Held out: other filings, other trials, other SAR seeds and the official test splits; never trained or tuned on. Thresholds were set on the dev split. TabFact's claims are human-written and the TAT-QA answers are real question-answer pairs over real report tables, so those two sets are the closest to real use; the other three use planted errors from our generator. "Caught" is recall on contradicted claims; "false flags" is the share of supported claims called contradicted. The LLM judge is Qwen3.8-27B (temperature 0), on a random sample of each set.
| Set | System | n | Accuracy | Caught | False flags |
|---|---|---|---|---|---|
| Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits) | code only | 3000 | 59.8% | 20.9% | 0.1% |
| Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits) | this model | 3000 | 97.9% | 97.8% | 2.6% |
| Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits) | code, then this model | 3000 | 94.3% | 96.2% | 1.7% |
| Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits) | Qwen3.8-27B judge (random sample) | 400 | 78.2% | 85.3% | 27.4% |
| Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits) | v1, internal ModernBERT-base (not released), for reference | 3000 | 97.5% | 96.7% | 2.3% |
| Planted errors, clinical-trial results (held-out CT.gov trials) | code only | 520 | 51.9% | 0.0% | 0.0% |
| Planted errors, clinical-trial results (held-out CT.gov trials) | this model | 520 | 96.9% | 93.3% | 0.4% |
| Planted errors, clinical-trial results (held-out CT.gov trials) | code, then this model | 520 | 95.6% | 92.9% | 0.4% |
| Planted errors, clinical-trial results (held-out CT.gov trials) | Qwen3.8-27B judge (random sample) | 178 | 86.5% | 77.9% | 8.9% |
| Planted errors, clinical-trial results (held-out CT.gov trials) | v1, internal ModernBERT-base (not released), for reference | 520 | 90.4% | 91.1% | 11.3% |
| Planted errors, synthetic bank ledgers (held-out seeds) | code only | 2727 | 96.0% | 98.9% | 0.0% |
| Planted errors, synthetic bank ledgers (held-out seeds) | this model | 2727 | 99.6% | 98.9% | 0.0% |
| Planted errors, synthetic bank ledgers (held-out seeds) | code, then this model | 2727 | 96.0% | 98.9% | 0.0% |
| Planted errors, synthetic bank ledgers (held-out seeds) | Qwen3.8-27B judge (random sample) | 200 | 78.5% | 55.3% | 5.4% |
| Planted errors, synthetic bank ledgers (held-out seeds) | v1, internal ModernBERT-base (not released), for reference | 2727 | 99.6% | 98.9% | 0.0% |
| TabFact test, human claims with a digit | code only | 1999 | 22.6% | 0.1% | 0.0% |
| TabFact test, human claims with a digit | this model | 1999 | 73.8% | 68.1% | 20.8% |
| TabFact test, human claims with a digit | code, then this model | 1999 | 72.6% | 60.7% | 16.2% |
| TabFact test, human claims with a digit | Qwen3.8-27B judge (random sample) | 86 | 83.7% | 81.6% | 12.5% |
| TabFact test, human claims with a digit | v1, internal ModernBERT-base (not released), for reference | 1999 | 65.5% | 61.6% | 30.8% |
| TAT-QA test gold answers restated as sentences by Qwen3.8-27B | code only | 800 | 32.3% | 4.3% | 0.5% |
| TAT-QA test gold answers restated as sentences by Qwen3.8-27B | this model | 800 | 79.4% | 87.3% | 28.2% |
| TAT-QA test gold answers restated as sentences by Qwen3.8-27B | code, then this model | 800 | 79.7% | 86.3% | 26.5% |
| TAT-QA test gold answers restated as sentences by Qwen3.8-27B | Qwen3.8-27B judge (random sample) | 300 | 60.0% | 40.4% | 18.2% |
| TAT-QA test gold answers restated as sentences by Qwen3.8-27B | v1, internal ModernBERT-base (not released), for reference | 800 | 41.4% | 47.0% | 21.0% |
v2 (ModernBERT-large, 2 epochs, 512 tokens) is more accurate than v1 on four held-out sets and level on the synthetic ledgers; it has more false flags on the TAT-QA set (28.2% vs 21.0%) and is about 3.4 times slower on a CPU. On free-form claims it closed most of the gap: TAT-QA answers restated in plain words 79.4% (v1 41.4%; Qwen3.8-27B 60.0% on a 300-claim sample where this model is at 79.0%; Qwen3.8-27B also wrote these sentences), TabFact human claims 73.8% (v1 65.5%; Qwen3.8-27B 83.7% on the same 86-claim sample where this model is at 67.4%). Counting and superlatives remain the LLM's ground. Treat it as a cheap first reader for number-bearing sentences that cite their table, not as a general fact checker.
Limitations
- Trained on planted errors. Its strength is the error kinds our generator plants; accuracy on real drafting errors is not measured. Planted financial sets carry some label noise from table quirks (undeclared units, oddly formatted cells).
- Free-form claims are weaker: TabFact human claims 73.8% with 20.8% false flags; TAT-QA plain-word answers 79.4% with 28.2% false flags (higher than v1's 21.0%).
- A learned bias we found: in our clinical-trial training data, "no one in group X died" appeared only as a planted error, so the model tends to call such sentences contradicted. Check "no one" and "none" claims with code, not with this model.
- Reads English only; tables of up to about 10 rows are shown to it (longer tables are cut to the rows the code and the claim touch).
- It checks claims against the evidence given, not against the world. It does not handle dates as well as amounts and percentages.
- It flags and ranks; it should not clear anything on its own in a regulated workflow.
Intended use
A pre-filter and second reader for number-bearing sentences in drafts that cite their tables: financial narratives (MD&A, reports), clinical-trial lay summaries, investigation narratives. It flags and ranks; a person decides. It is not a compliance tool, it does not check claims against the world (only against the evidence given), and it should not clear anything on its own in a regulated workflow. For clinical text it checks that a sentence matches the trial's reported tables; it is not medical advice and must not be used to make medical or treatment decisions.
Model
ModernBERT-large (answerdotai/ModernBERT-large, Apache-2.0) with three heads on the first token and the tokens: verdict (3 classes), error type (6), evidence pointer (per token). 512-token input: the claim, then the code tier's findings, then the evidence with cell ids. 2 epochs on 81,032 pairs, lr 3e-5, batch 32, on one rented A100 (about 36 minutes). Dev split: accuracy 91.8%, error type 97.6%, pointer 96.6%. Temperature 1.25 and the two thresholds were set on dev. ONNX export parity: max probability difference 5e-5, 20 of 20 argmax agree.
Files: model.safetensors (fp32 encoder plus heads), config.json (the encoder), tokenizer files, checker.json
(labels, temperature, thresholds), modeling_checker.py, numparse.py, onnx/model.onnx, eval_summary.json,
LICENSE, NOTICE, SHA256SUMS.
Licence
Apache-2.0 for the weights and code in this repository. The base model is Apache-2.0 (ModernBERT, Answer.AI and LightOn). The training data sources above ask for attribution; it is given here and in NOTICE.
Citation and contact
Decosa, "decosa-number-checker-modernbert-large (v2)", 2026, decosa.ai. Built with planted errors from Decosa's own pipelines; the generator and the held-out sets are not released.