decosa-grounding-modernbert-large (v1)

A grounding checker you can use commercially. Given a source passage and one claim sentence, it returns:

• a verdict: supported, contradicted or not_in_source, with a probability for each (p_flag = 1 - p(supported));
• the likeliest error type: entity swap, number change, negation, unsupported addition, overgeneralisation, temporal shift, or absent (on topic, but not in the source). This is a hint for reviewers, not a verdict;
• evidence: the source tokens that support or contradict the claim, merged into character spans (or scored per sentence when you pass sentence boundaries);
• the claim words it thinks are wrong or added.

It runs on a CPU. On a shared AMD Ryzen 9 9950X3D (8 threads, fp32, other jobs running) one sentence takes 0.72 s at p50 and 1.14 s at p90. On an A100 it takes 23 ms. It was built as the first stage in front of an LLM grounding judge (Qwen3.8-27B, about 2 s and ~1,100 prompt tokens per sentence). It doesn't replace a person who reads the source.

Why it exists: the strongest open grounding checkers are non-commercial (Bespoke-MiniCheck-7B and Lynx are CC BY-NC 4.0), or were trained on data with non-commercial or unclear terms (ANLI inside MiniCheck-DeBERTa/FT5 and AlignScore, MS MARCO inside AlignScore, RAGTruth inside LettuceDetect). This model starts from an Apache-2.0 encoder. Every training source is public domain, CC0, CC BY 4.0, reusable under Commission Decision 2011/833/EU, or synthetic.

Results

Held-out test sets are split by source document, so no test document is seen in training, not even as a distractor. Thresholds and the cascade band were set on the dev split only. "LLM" is Qwen3.8-27B running the Decosa grounding block's own judge prompt, on a stratified sample of 720 test pairs (120 per domain, 40% supported).

On SummEdits an item is flagged when any summary sentence is flagged (model: p_flag >= 0.646, the dev threshold for 5% false flags; LLM: any verdict other than supported, partial included). Balanced accuracy by domain, model vs LLM: billsum 0.70 vs 0.73, ectsum 0.90 vs 0.80, news 0.77 vs 0.80, podcast 0.60 vs 0.90, qmsumm 0.67 vs 0.87, sales call 0.87 vs 0.93, sales email 0.83 vs 0.87, samsum 0.73 vs 0.83, scitldr 0.77 vs 0.77, shakespeare 0.67 vs 0.77. On this external set the model is clearly weaker than the LLM judge, so use it as a cheap first pass, not as the only check.

Catch rate by error type on the 720-pair sample (model vs LLM): number change 100% vs 98.3%, temporal shift 96.7% vs 88.3%, overgeneralisation 90.0% vs 86.7%, negation 98.3% vs 100%, unsupported addition 98.3% vs 100%, entity swap 93.9% vs 95.5%, absent 93.9% vs 93.9%. With 60-66 errors per type, one miss is 1.5-1.7 points, so these per-type gaps are within noise. Evidence sentences: precision 0.85, recall 0.95. Wrong-word spans in caught errors: recall 0.91, precision 0.54 (it marks too many words).

Intended use

• Checking AI-written summaries, answers and drafts against the source they cite: clinical notes, filings, contracts, drug labels, lay summaries, product labels, rules and notices. Split the output into sentences and check one at a time.
• As the first stage of a cascade. Clear sentences are passed or flagged, and the uncertain middle goes to an LLM judge or a person.

Not for medical, legal or financial decisions. On clinical text it is a reviewing aid that points a person at sentences to re-read; it does not check whether a note is clinically correct.

Also not for: world-knowledge fact checking (it only asks "does this source say so?"), long sources in one call (it reads about 768 sub-word tokens of source per claim, so pick the relevant window first), languages other than English, or any decision about a person without human review. A supported verdict means that no error of the trained kinds was found. It does not mean the claim is true.

Limitations

• Trained and tested on planted errors. The claims and the edits were written by an LLM (Qwen3.8-27B) and not reviewed by people. The labels therefore carry that LLM's mistakes: some "supported" training claims are not fully supported. In-domain numbers are optimistic, and the external set (SummEdits) is the better guide to real-world behaviour: there it catches 83% of inconsistent summaries with 33% false flags, below the LLM judge (92% / 27%).
• Weakest on conversational sources (meeting transcripts, podcasts, dialogue; balanced accuracy 0.60-0.73 on SummEdits) and on literary text. It was trained on none of these.
• One claim sentence against one window. It doesn't combine evidence across windows, and it has no partial verdict.
• Synthetic clinical notes and filings are fictional and cleaner than real records. Abbreviations, templated EHR text and tables haven't been tested.
• Wrong-word spans are over-wide (precision 0.54).

Training data (every source and its licence; read 27 Sep 2026)

Plus rule-based plants: number edits with exact spans, source sentences copied as easy positives, and hard negatives (a supported claim from another document of the same domain). Not used: ANLI (CC BY-NC), MS MARCO (non-commercial research terms), RAGTruth (built on MS MARCO passages), HaluBench (CC BY-NC 2.0), LLM-AggreFact (CC BY-ND 4.0, and gated terms not accepted), and any NC checkpoint or its outputs. No training or evaluation data is included in this repository; the generator and the held-out sets aren't released. SummEdits (Salesforce/summedits, CC BY 4.0) was used for evaluation only, never for training or threshold fitting.

Model

ModernBERT-large (answerdotai/ModernBERT-large, rev 45bb4654, Apache-2.0) reads claim [SEP] source, up to 768 tokens. Heads: a 3-way verdict softmax (CLS + mean pooling), an error-type softmax over 8 classes, and a per-token softmax over O / EVID / ERR. Loss: verdict CE + 0.3 x type CE + 0.5 x token CE (weights 1/2/4). Trained on 23,056 pairs for 3 epochs (2,163 steps, batch 32, lr 3e-5 for the encoder and 1e-4 for the heads, bf16 autocast) on one rented A100 80 GB, in 14.5 minutes. The checkpoint was chosen on dev (flag AUC 0.985, macro-F1 0.932).

Files: model.safetensors (fp32), config.json (the encoder), tokenizer files, gm_config.json (labels, thresholds, max length), gm_model.py (the heads), usage.py, eval_summary.json (all numbers above), SHA256SUMS.

Usage

pip install torch transformers safetensors
python usage.py . "Adults: take 10 mg twice daily with food." "Take 20 mg twice daily with food."
# contradicted  p_flag=0.9946  number_change  cascade=flag   (CPU, fp32)

Licence

Apache-2.0 for the weights and code here. The base model is Apache-2.0 (Answer.AI / LightOn). The data attributions are in NOTICE.

Contact

Decosa, decosa.ai. The numbers above are in eval_summary.json.