decosa-note-detail-checker-modernbert-large (v1)

A note-detail checker for AI-drafted clinical notes. It checks one detail of a note sentence against the visit transcript and answers same, changed or absent: is the value the same as what was said, changed from it, or not in the transcript lines given? The detail can be a drug, dose, frequency, route, date, duration, side or number. The model also tags the transcript tokens it used as evidence.

It is a documentation-QA aid that points a reviewer at details to re-check before a note is signed. It checks a note against a conversation. It does not assess care or say whether anything is clinically right. It is not medical advice and not a medical device.

In our pipeline it settles a language-model judge's "a detail is not in the visit" flags. On held-out notes it raised the share of changed details caught from 72% to 94%, with no new false flags on faithful notes. It runs on a CPU.

Results (held out)

Test set. 50 PriMock57 test consultations, never used in training or threshold setting. The reference transcripts come from the dataset's per-speaker annotations, so they contain no speech-recognition errors. Each visit has a synthetic US-style scribe note in which every sentence cites transcript lines; these were written by a different model family from the judge. Each note has one altered copy with exactly one changed detail. A second, larger set of 200 changed-detail plants on the same 50 visits was written blind by a separate model agent (8 detail types) and was never trained on.

The judge and the rule. The judge is a general open model, Qwen3.8-27B, that checks each sentence against the whole transcript. The rule was frozen on dev before the test run and the test ran once. A sentence the judge flags as PARTIAL ("a detail is not in the visit") becomes a "changed detail" error when one of its details has p(changed) >= 0.5. Sentences the judge calls SUPPORTED are never upgraded. On dev, upgrading them added 4-5 false flags in 175 faithful sentences.

By detail type on the 200 (judge alone, then with this model):

Twelve plants were still missed. In all twelve, the judge flagged the sentence for review and the model read the detail as same or absent (p(changed) below 0.5).

Alone, with no language model. Here the transcript window is picked by word overlap only, and a flag needs p(changed) >= 0.9:

• 33 of the 50 changed details caught (66%), with 3 of 1,135 faithful sentences flagged (0.26%);
• 121 of the 200 extra plants caught (60.5%);
• with the judge's cited lines added to the window, 39 of 50 caught, but 14 of 1,135 faithful sentences flagged.

Against a frontier proofreader. A closed frontier model proofread 40 of the test notes blind (20 with a changed detail, 20 faithful), working as a careful clinician would before signing:

ACI-Bench test set 1 (39 of 40 encounters; one note is over our 80-sentence limit). These are in-person US visits with dictated exams and human-written notes. The transcripts keep the speech-recognition and speaker-tag errors that the dataset includes on purpose, and the model was trained on ACI-Bench train only.

• 78 changed-detail plants, written blind by a separate model agent: judge alone caught 71 (91%), judge + this model 72 (92.3%). The judge is already strong on these visits, so there is little left for this model to add.

• The human notes as written (1,469 checked sentences) are not a false-flag test, because clinicians write things that were never said aloud. The judge raised error flags on 97 sentences (6.6%), and this model turned 9 more "detail not in the visit" flags into errors. We read them by hand:

  • 2 were real conflicts: "the past several months" against "six weeks", and 40 mg against "four milligrams".
  • 3 were details never said aloud: ages worked out from a date of birth, and a medicine strength.
  • 4 were false upgrades: "type 1" against the recogniser's "type i"; "yesterday around dinnertime" against "last night around supper time"; "04/2022" against "starting in April"; and "right arm" against "my arm hurts right here".

  So on real transcripts it adds some false error flags, about 4 in 1,500 sentences here.

Speed. On CPU, a note took a median of 7.1 s (12 notes, about 18 details per note, 8 threads of a shared machine under other load, fp32).

Intended use

• Quality assurance of AI-drafted clinical notes (ambient scribes, summarisers) against the visit transcript, for numeric and medication details. Use it to point a clinician or reviewer at details to re-check before signing, and to monitor a scribe's error rate over many notes.
• As a second stage behind a language-model judge, to settle that judge's "detail not in the visit" flags.

Out of scope

• Medical advice, diagnosis or treatment decisions, or deciding whether a note is clinically correct.
• Judging or scoring clinicians.
• Invented facts, wrong speakers, invented exam findings and omissions. It checks only a marked detail against the given lines, so these need a judge or a person.
• Checking notes with no transcript. Checking against summaries or other notes instead of the conversation.
• Languages other than English. Specialties, inpatient care and multi-party visits (such as interpreters or family) beyond what the datasets contain, all of which are untested.

Limitations

• Test notes are synthetic. The model was trained and tested on synthetic notes with clean single swaps. It has not been measured on real scribe output with real errors, which are subtler and tend to cluster.
• Silver labels. Training labels were produced by code, not by clinicians, so some "same" pairs are wrong where a note paraphrased loosely.
• Limited coverage. English primary care: remote UK consultations (PriMock57), in-person US visits (ACI-Bench) and synthetic visits.
• Transcript quality matters. Speech-recognition slips ("type i" for "type 1") and loose paraphrase of times can read as "changed".
• Recall depends on the judge. In the combined rule, a changed detail that the judge calls SUPPORTED stays missed.
• Detail finding is not included. The results above used our rule-based detail extractor (a medicine lexicon plus patterns for doses, frequencies, dates, durations, sides and numbers) and our window picker. Those rules are not in this repository, and details they don't recognise were not checked.
• Very short windows read as "absent". In spot checks while preparing this release, a 3-line window returned absent (p >= 0.97) whether the value matched or had been changed. The same content in a 5- or 8-line window gave the right answer (p >= 0.999). Windows in a 60-pair dev sample had 2-9 lines (median 7), so pass at least 5 lines.
• The evidence head is a hint. It sometimes points at the wrong line. Our product quotes it only when the judge cited the same line.

Input format

• Claim. The note sentence with the detail wrapped in [[ ]], prefixed by its type, for example [frequency] Ibuprofen 400 mg [[three times a day]] after food.
• Source. The transcript lines to read, one per line, as <n> [<Speaker>] <text>.
• Length. Up to 512 sub-word tokens in total; the source is truncated after that. Pass at least 5 lines (see Limitations).
• Window choice. We pick the lines by word overlap with the sentence, add any line that says a value of the same kind, and add the judge's cited lines when there is a judge.

Training

• Starting point. decosa-grounding-modernbert-large at revision 5482aa8b, which is ModernBERT-large (Apache-2.0) with three heads. Here the verdict head reads same / changed / absent, the error-type head reads the kind of change (auxiliary), and the token head tags transcript evidence.
• Pairs. 26,144 training pairs (11,914 same, 7,020 changed, 7,210 absent) and 4,338 dev pairs (1,975 / 1,155 / 1,208). Training pairs by detail type: frequency 5,158, drug 4,900, number 3,575, dose 3,491, date 3,455, duration 2,117, side 2,007, route 1,441.
• Labels, by code. "Same" means the detail's value is found, after normalising number words, units and misspelt medicines, among values of the same kind in its window. "Changed" means a same detail was swapped by rule for another value of its kind. "Absent" means the evidence lines were removed (sometimes leaving other values of that kind as distractors), or a dose, frequency or side was inserted.
• Visits. Training used 237 synthetic visits (21,527 pairs) and 61 ACI-Bench train encounters (4,617 pairs). Dev used 21 synthetic visits, 19 ACI-Bench valid encounters, 5 ACI-Bench train encounters and the 7 PriMock57 dev consultations. No PriMock57 test consultation and no ACI-Bench test set is in training or dev.
• Settings. Verdict cross-entropy + 0.2 x type cross-entropy + 0.3 x token cross-entropy. 3 epochs, batch 16, learning rate 2e-5 for the encoder and 1e-4 for the heads, maximum length 512, bf16 autocast, one GPU, about 9 minutes.
• Checkpoint. Step 4,000 was chosen on dev: flag AUC 0.999 and 3-way macro-F1 0.964, on dev pairs that are mostly synthetic.
• Runs. Three runs, changed only on what dev showed. The first called faithful details "changed" when the window missed the evidence but held another value of the same kind. The second fixed that with value-aware windows and distractor "absent" examples. The third fixed a number normaliser that had dropped spelled-out numbers from the "same" class. The third run is released.

Training data and licences (read 28 Sep 2026)

No training or evaluation data is included in this repository. The generators, the pair builder and the planted test sets are not released. The training pairs contain no text written by closed models. Closed models wrote only evaluation notes and plants (the dev notes used to fix the rule, and the test notes), and none of those were trained on.

Model files

• model.safetensors (fp32, 1.59 GB);
• config.json (the encoder) and the tokenizer files;
• gm_config.json (labels, thresholds, provenance) and gm_model.py (the heads);
• usage.py, eval_summary.json (every number above) and SHA256SUMS.

Run it on CPU

pip install torch transformers safetensors
git clone https://huggingface.co/decosaai/decosa-note-detail-checker-modernbert-large && cd decosa-note-detail-checker-modernbert-large
W=$'1 [Clinician] good morning how are you doing today\n2 [Patient] not too bad thanks\n3 [Clinician] so what brings you in\n4 [Patient] i have had this cough\n5 [Clinician] okay let me ask a few questions\n13 [Clinician] and are you still taking the metformin\n14 [Patient] yes five hundred milligrams twice a day with meals\n15 [Clinician] okay good'
python usage.py . "[dose] Metformin [[1000 mg]] twice daily with meals." "$W"
python usage.py . "[dose] Metformin [[500 mg]] twice daily with meals." "$W"
python usage.py . "[frequency] Metformin 500 mg [[three times a day]] with meals." "$W"

changed  p(same)=0.0001  p(changed)=0.9998  p(absent)=0.0001  -> changed detail (standalone threshold)
evidence lines: 14
same  p(same)=0.9999  p(changed)=0.0000  p(absent)=0.0001  -> no change flag
evidence lines: 14
changed  p(same)=0.0000  p(changed)=0.9999  p(absent)=0.0000  -> changed detail (standalone threshold)
evidence lines: 14

(CPU, fp32. These are illustrative, not a measurement. The same three lines 13-15 passed on their own came back absent; see Limitations.)

Thresholds are in gm_config.json. Use upgrade_partial (0.5) only together with a judge's flag on the sentence, and alone (0.9) when there is no judge.

Licence

The weights and code here are Apache-2.0. The base model and the grounding model are Apache-2.0. The data attributions are in NOTICE.

Contact

Decosa, decosa.ai. The numbers above are in eval_summary.json.