decosa-citation-support-modernbert-large (v1, prototype)

A citation-support checker for legal briefs. You give it one sentence from a brief that cites a case and passages from the cited opinion. It returns p_flag = 1 - p(supported): how likely it is that the passages do not support the sentence as written, for example a misstated holding.

It is a reviewing aid for cite-checking. A flag means "a person should read this opinion against this sentence". It is not legal advice and it does not decide what a case holds.

It is a fine-tune of our grounding model decosa-grounding-modernbert-large on 732 brief-sentence / opinion-passage pairs. It runs on a CPU at about 0.9 s per pair.

Read the results by subset. Most of what it catches on the benchmark are short holdings written by language models. On holdings that the benchmark's authors altered inside real briefs, it catches far fewer (see the second table).

Results (held out)

Test set: 501 pairs built from 242 of the 390 rows of the LePhantomCite eval split. 129 pairs are misstated holdings and 372 are supported. None of these rows was seen in training or in threshold setting. The other 148 rows gave no usable pair, because the case wasn't found or the sentence wasn't parsed. Thresholds were set on the validation split only (below). "Judge" is a general open model, Qwen3.8-27B, asked whether the passages support the sentence. The test ran once.

By subset. The test pairs come from two kinds of rows:

So the headline 80 of 129 is mostly the second subset (68 of 78). On altered holdings inside real briefs, the model paired with the judge catches 12 of 51 (24%) at 4 false flags in 359 supported sentences (1.1%). That is lower recall than the judge's "contradicted" verdict alone (21 of 51) but with far fewer false flags (4 against 21). Use it to settle a judge's doubts with fewer false alarms. It is not a detector you can rely on alone.

By how much of the opinion it saw (model alone at 0.304). When the cited opinion was short enough to pass whole, it caught 77 of 110 with 13 false flags in 253 (AUC 0.917). When only retrieved passages were passed, it caught 7 of 19 with 6 false flags in 119 (AUC 0.784). If retrieval misses the relevant passage, the model can't tell.

Validation (141 pairs split by row from the training pool: 21 misstated, 120 supported). AUC was 0.679 before fine-tuning and 0.849 after the first epoch. Later epochs scored lower (0.816, 0.789, 0.809, 0.766), so the first epoch was kept. The thresholds come from this split: 0.304 gave 6 of 120 false flags (5.0%) and caught 9 of 21, and 0.581 gave 3 of 120 (2.5%) and caught 7 of 21.

Speed. On CPU, the 501 test pairs took 456-469 s on 8 threads of a shared desktop machine with other jobs running, about 0.9 s per pair (fp32).

Intended use

• Cite-checking a brief, your own or the other side's. First find the cited opinion and pull the passages that bear on the sentence. Then score the sentence against them, and send flagged sentences to a lawyer who reads the opinion.
• As a second opinion on an LLM judge's "unsupported" or "contradicted" verdicts, which cuts false alarms.

Out of scope

• Legal advice, or deciding what a case holds. Deciding whether a filing is sanctionable, or judging a lawyer or a litigant.
• Whether a case exists, whether the name matches the reporter citation, or whether a quotation is verbatim. Those need lookups and string checks, not this model.
• Pin cites. It reads passages from anywhere in the opinion, not the cited page, so a wrong pin page is invisible to it. For comparison, the full cite-checking pipeline it was built for caught 2 of 55 wrong pincites on this benchmark.
• Statutes, regulations and secondary sources (it was trained on case citations only). Non-US law, and languages other than English.
• Long opinions without retrieval: it reads about 768 sub-word tokens (sentence plus passages) and truncates the rest.

Limitations

• Small and narrow training set. 732 training pairs, 117 of them misstated. One seed, and one validation split of 141 pairs. The model overfits after one epoch.
• What "misstated" means here. It means either the benchmark authors' edits to real briefs or holdings that language models wrote in a 2024 study. Neither is a sample of real misstatements in filed briefs, and we have not measured the model on those.
• What "supported" means here. Supported pairs are sentences from real briefs with no injected error, assumed correct. Some are not, so a few "false flags" may be real problems in the original briefs.
• Only the verdict head was fine-tuned. The error-type head and the evidence-token head come from the grounding model unchanged, and the encoder under them has moved. They were not trained or tested for this task. Ignore them.
• Only "supported" is meaningful. The model was trained as supported against misstated, with misstated in the "contradicted" slot. "not_in_source" was never a training target, so read only 1 - p(supported).
• US federal and state case law in English. Opinion text comes from the Caselaw Access Project and from court PDFs, so OCR noise is present.

Training

• Starting point. decosa-grounding-modernbert-large at revision 5482aa8b. That model is ModernBERT-large (Apache-2.0) with a 3-way verdict head, an error-type head and a token head.
• Pairs. Each pair is one sentence from a brief excerpt that cites a case (the claim), plus up to 6,000 characters of that case's opinion text (the source). The source is either the whole opinion or the passages our retrieval picks for the sentence. The pairs were built with the same parser and retrieval that the cite-checker uses.
• Labels. "Misstated" means the benchmark marked the sentence as a content misrepresentation. "Supported" means the excerpt has no injected error of any kind. Sentences on excerpts with other injected errors were skipped.
• Pool and split. 873 pairs came from 362 rows of the LePhantomCite aux_train split: 138 misstated, 735 supported, 36 of them from the LLM-written subset. They were split by row, 85/15, into 732 training pairs and 141 validation pairs.
• Settings. Cross-entropy on the verdict head with class weights 1 / 3 / 1, batch 8, learning rate 1.5e-5 for the encoder and 1e-4 for the pooler and verdict head. Maximum length 768, bf16 autocast, five epochs run, first epoch kept. It trained on one rented A100 80 GB at about 20 s per epoch.

Training data and licences (read 28 Sep 2026)

No training or evaluation data is included in this repository, and the pair builder isn't released. Nothing in the training data was written by Decosa or by our own models: the claims are from the benchmark and the sources are court opinions.

Model files

ModernBERT-large reads claim [SEP] source, up to 768 tokens, under the grounding model's heads. The files are:

• model.safetensors (fp32, 1.59 GB);
• config.json (the encoder) and the tokenizer files;
• gm_config.json (labels, thresholds, provenance) and gm_model.py (the heads);
• usage.py, eval_summary.json (every number above) and SHA256SUMS.

Run it on CPU

pip install torch transformers safetensors
git clone https://huggingface.co/decosaai/decosa-citation-support-modernbert-large && cd decosa-citation-support-modernbert-large
python usage.py . "<the brief's sentence>" "<passages from the cited opinion>"

For example, with a passage from Miranda v. Arizona, 384 U.S. 436, 444 (1966):

P="Our holding will be spelled out with some specificity in the pages which follow but briefly stated it is this: the prosecution may not use statements, whether exculpatory or inculpatory, stemming from custodial interrogation of the defendant unless it demonstrates the use of procedural safeguards effective to secure the privilege against self-incrimination."
python usage.py . "The prosecution may not use statements stemming from custodial interrogation unless it shows procedural safeguards effective to secure the privilege against self-incrimination." "$P"
python usage.py . "Miranda holds that statements made during custodial interrogation are admissible whenever the suspect was not formally arrested." "$P"

p_flag=0.0007  no flag  (flag at 0.304; strict 0.581)
p_flag=0.9827  FLAG: read the opinion against this sentence  (flag at 0.304; strict 0.581)

(CPU, fp32. These are two illustrative sentences, not a measurement; the tables above are the measurement.)

To use it in code, load it as usage.py does. Batch pairs with truncation="only_second", and flag at gm_config.json thresholds.flag (0.304) or thresholds.flag_strict (0.581).

Licence

The weights and code here are Apache-2.0. The base model and the grounding model are Apache-2.0. The data attributions are in NOTICE.

Contact

Decosa, decosa.ai. The numbers above are in eval_summary.json.