ClarioScope insurance extractor v1

A 125M-parameter RoBERTa-base fine-tune that extracts structured insurance and billing fields from inbound patient text. It tags spans across 12 field types — carrier, plan type, member ID, group number, claim ID, copay, deductible, billed amount, and more — and produces output that downstream billing systems can ingest as JSON. This is model 3 of the ClarioScope SLM Suite — a three-model intake intelligence pipeline for healthcare practices.

TL;DR

Note on benchmark scope. This release benchmarks against GPT-4o only. Anthropic API credit was exhausted during the development cycle before the Claude Haiku 4.5 / Sonnet 4.6 comparisons could be run. A subsequent revision of this card will add the Anthropic numbers.

Why this model exists

Patient inquiries about insurance, billing, and prior authorizations have a different shape from the rest of intake. The information value is in the structured fields — carrier name, member ID, group number, claim reference, copay amount — not in the surrounding prose. A practice's billing system can act on {"carrier": "Aetna", "plan_type": "PPO", "member_id": "AET-998-2210"} directly; it cannot act on "hi can you check if my Aetna PPO covers..." until those fields are extracted.

Frontier LLMs do this extraction well, with one consistent caveat — they cost real money per call, add ~1 second of latency, and send patient text to a third party. This model is the self-hosted alternative: 26× lower latency, $0 per inference after training, and patient text never leaves the host. It trails GPT-4o by 17 macro F1 points on aggregate, but it ties GPT-4o on the four fields with the most volume (MEMBER_ID, CARRIER, CLAIM_ID, SUBSCRIBER_NAME).

This is model 3 of the ClarioScope SLM Suite:

1. Intent classifier (clarioscope-intent-deberta-v1) — what does this message want?
2. PHI detector (clarioscope-phi-deberta-v1) — where is protected information in this message?
3. Insurance extractor (this model) — what billing-relevant structured data is in this message?

The 12 field types

The model outputs BIO token labels (25 total: O + 12 × {B-, I-}), which downstream code converts back into character-offset spans and then into a JSON object.

Architecture

Standard RoBERTa-base encoder with a token-classification head: a linear layer over each token's contextualized representation, producing 25 logits per token. All 125M parameters are fine-tuned. Training uses fp32 (not mixed precision — see the PHI detector's card for the NaN-gradient story that motivated this), adamw_torch optimizer, max_grad_norm=1.0, an explicit classifier-head re-init (std=0.02 normal, zero bias), batch size 8, sequence length 256, learning rate 1e-5 with cosine schedule and 10% warmup, weight decay 0.01, and early stopping (patience 2 on macro F1). Five epochs run in ~10 minutes on a single RTX A4000.

Training data — synthetic, transparent about it

All training and evaluation data is synthetic. There is no real patient billing data in this model or its evaluation.

Training set (8,086 examples after cleanup, from 8,120 originally). Generated via the OpenAI API (gpt-4o-mini-2024-07-18, JSON-object response format, temperature 1.0) across 12 field types × healthcare practice types × channels. The generation prompt enforces a 40/40/20 realism mix (polished / casual / messy) and channel-specific scaling (SMS messiest, voicemail second messiest, email and web forms cleaner).

Data cleanup. Same cue-word noise pattern observed in the PHI detector: gpt-4o-mini sometimes returns entity texts that include the cue word ("member ID AET-998-2210" instead of "AET-998-2210", "copay $35" instead of "$35"). clean_data.py in the repo strips these cue-word prefixes and re-locates the cleaned text in the source. 1,393 entities (~7.4% of all spans) were normalized during cleanup; 256 entities became empty after stripping and were dropped.

Test set (672 examples). Generated by Claude Haiku 4.5 with a deliberately different prompt style and abbreviation set (w/, &, ins, mbr, grp versus the train prompt's more formal style). This cross-generator split mitigates benchmark leakage from training the model on one generator's style.

Anti-leakage validation. The cross-generator split produces a large val/test gap (val macro F1 0.96; test macro F1 0.79). The val set comes from the same gpt-4o-mini distribution as training; the test set comes from a different generator. The 17-point gap is the cost of generalizing across generators — and a useful proxy for the gap that would exist between this model and a real-world distribution. Val numbers from same-generator splits routinely overstate real-world generalization on tasks like this.

Benchmark — entity-level F1, span boundaries matter

Evaluated on the 672-example held-out test set using seqeval, which requires both the entity type AND the exact span boundary to match for a true positive.

GPT-4o wins on aggregate, but the per-entity breakdown shows the gap is concentrated in a few low-frequency fields.

Per-field breakdown

High-volume fields where fine-tune is competitive:

SUBSCRIBER_NAME is essentially tied with GPT-4o (0.89 vs 0.91). The four highest-volume fields (CARRIER, MEMBER_ID, CLAIM_ID, BILLED_AMOUNT) collectively cover ~70% of the test entities, and the fine-tune is within 5–13 points of GPT-4o on each.

Mid-volume fields with moderate gaps:

GROUP_NUMBER has a 19-point gap despite 216 training-supported examples in test. The format variance for group numbers is wide (4421, 001428, GRP-882044, Group #44210) and the cross-generator split exposes the train-set format bias.

Low-volume / low-F1 fields:

AUTH_NUMBER is the headline weakness — 0.30 vs 0.99. The training set has only 770 AUTH_NUMBER spans total, and the format space is wide (PA-4421, auth #998-2210, AUTH998212, etc.). A v2 with more AUTH_NUMBER coverage in training would likely close most of this gap.

RELATIONSHIP is a hard category for both models — short string ("self", "spouse", "child"), often overlapping with other entity contexts, with a tight span boundary that's easy to miss by one token.

Recommended production pattern: hybrid

For a billing pipeline that processes inbound patient messages:

1. Run this model first on every message. Captures CARRIER, MEMBER_ID, CLAIM_ID, SUBSCRIBER_NAME with near-frontier F1 at ~45 ms / message on CPU.
2. Add regex for high-value structured patterns: dollar amounts (\$[\d,]+(?:\.\d{2})?), member ID format checks specific to your top carriers (Aetna IDs, BCBS IDs all follow distinct patterns).
3. Use GPT-4o as a fallback for messages where the fine-tune is uncertain or for AUTH_NUMBER / PLAN_TYPE detection. The fallback should fire on ~10–20% of messages, not 100%.

This pattern is the same architecture recommended for the PHI detector. The fine-tune does the bulk-volume linguistic work; frontier APIs handle the long tail. Together they cost an order of magnitude less than running frontier on every message.

How to use

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

model_id = "raihan-js/clarioscope-insurance-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
model.eval()

text = ("Hi, I'd like to verify my coverage before tomorrow's appointment. "
        "My carrier is Aetna PPO, member ID AET-998-2210, group #4421. "
        "I'm the subscriber. My copay should be $35 — is that right?")

enc = tokenizer(text, return_offsets_mapping=True, return_tensors="pt", truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0].tolist()
with torch.no_grad():
    pred_ids = model(**enc).logits.argmax(dim=-1)[0].tolist()

id2label = model.config.id2label
spans = []
i = 0
while i < len(pred_ids):
    label = id2label[pred_ids[i]]
    if label.startswith("B-"):
        ent_type = label[2:]
        start = offsets[i][0]
        end = offsets[i][1]
        j = i + 1
        while j < len(pred_ids) and id2label[pred_ids[j]] == f"I-{ent_type}":
            end = offsets[j][1]
            j += 1
        spans.append({"text": text[start:end], "label": ent_type})
        i = j
    else:
        i += 1

# Convert to structured JSON for downstream billing system
extracted = {span["label"]: span["text"] for span in spans}
print(extracted)
# {'CARRIER': 'Aetna', 'PLAN_TYPE': 'PPO', 'MEMBER_ID': 'AET-998-2210',
#  'GROUP_NUMBER': '4421', 'RELATIONSHIP': 'subscriber', 'COPAY': '$35'}

Limitations

• All training and evaluation data is synthetic. No real patient billing data was used. Production deployment should include calibration against real-world inbound messages.
• AUTH_NUMBER is materially weaker than frontier (0.30 vs 0.99). Do not rely on this model alone for prior-auth extraction.
• Format brittleness on structured IDs. When a member ID, group number, or claim reference uses a format outside the training distribution, the model often produces a span boundary off by one token, which is a miss under strict matching.
• English only, healthcare practice domain only.
• RELATIONSHIP accuracy is moderate (0.70). The category overlaps with other named entities and short-span detection is fragile in cross-generator evaluation.
• Benchmark scope. Only GPT-4o was benchmarked in this release; Anthropic API credit was exhausted before Haiku 4.5 / Sonnet 4.6 could be run.

Intended use

A first-pass insurance / billing field extractor in a hybrid intake pipeline for healthcare practice software. Strongest on high-volume fields (CARRIER, MEMBER_ID, CLAIM_ID, SUBSCRIBER_NAME). Should be paired with regex matchers (for dollar amounts and known ID formats) and a frontier-API fallback (for AUTH_NUMBER and PLAN_TYPE edge cases).

Out-of-scope use

• Sole reliance for prior-auth processing. The model misses 70% of AUTH_NUMBERs on test data.
• Direct downstream billing without validation. Any extracted member ID, group number, or claim reference should be validated against the carrier's verification endpoint before being relied on for payment routing.
• Adversarial inputs. Not hardened against prompt injection or adversarial text crafted to manipulate extraction.

Citation

@misc{sikder2026clarioscope_insurance,
  author = {Sikder, Akteruzzaman Raihan},
  title  = {ClarioScope insurance extractor v1: a 125M-parameter RoBERTa fine-tune for structured insurance and billing extraction},
  year   = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/raihan-js/clarioscope-insurance-v1}},
}

Read more

The full ClarioScope SLM Suite writeup — methodology, cost ledger, and per-model interpretation — is on dev.to. Each model has its own post; the suite-level summary is at the GitHub profile github.com/raihan-js.

Author

Built by Akteruzzaman Raihan Sikder — then CTO of ClarioScope AI (2024–2026; the company has since been sunset and the suite is open source). Part of the broader ClarioScope SLM Suite (intent classifier, PHI detector, insurance extractor) — a three-model intake intelligence pipeline.