ModernBERT-large for full-conversation legal-guidance detection

Paper: Legal Guidance in the Wild | Code: GitHub

This checkpoint predicts whether a conversation's user is seeking legal guidance. It is the first stage of the full-conversation ModernBERT-large cascade from the Legal Guidance in the Wild study. Input contains chronological user messages only; assistant messages are excluded. The paired topic model is modernbert-large-v3-primary-topic-user-len4096-seed42.

This is a research routing model, not a legal-advice system. It must not be used to decide whether a person has a valid claim or needs professional counsel.

Labels

• False: no legal guidance sought
• True: legal guidance sought

Data

• Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
• Requested revision: main (latest at run time)
• Train fingerprint: 6ba4c2696e889276
• Validation fingerprint: 403fe118d76c8360
• Train / validation conversations: 1,632 / 290
• Input mode: chronological user messages only

Dataset access follows the linked repository's sharing settings. The fingerprints identify the exact loaded splits even if main later changes.

Training configuration

The checkpoint was selected by validation macro-F1, followed by guidance threshold selection. The latest complete run selected threshold 0.38.

Silver-validation results

The paired topic checkpoint obtains 0.716 accuracy and 0.574 macro-F1 on the 134 silver guidance-positive validation conversations. Gold-set results are reported below.

Gold-set results

Scored once on the 200 human-adjudicated gold conversations after all modeling decisions were frozen (paper, Tables 4 and 6; per-example predictions are in the GitHub repository).

Inference

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "AmirMohseni/modernbert-large-v3-seeks-guidance-user-len4096-seed42"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

# conversation: list of {"role", "content"} messages; only user turns are used.
user_only_conversation = "\n".join(
    f"User: {m['content']}" for m in conversation if m["role"] == "user"
)
inputs = tokenizer(user_only_conversation, return_tensors="pt", truncation=True,
                   max_length=4096)
logits = model(**inputs).logits
probability = logits.softmax(-1)[0, model.config.label2id["True"]].item()
seeks_guidance = probability >= 0.38

The threshold 0.38 is the frozen validation-selected operating point and is the value used for every reported result. The repository's class labels alone do not encode it, so an argmax over the two logits will not reproduce the cascade predictions.

Limitations

The checkpoint was trained on English-language public LLM interaction logs with silver labels, one source, and one seed. Inputs beyond 4,096 tokens are truncated. The dataset is jurisdiction-agnostic, may contain sensitive content, and is not representative of all legal-help seekers. Calibration and accuracy may shift across platforms, populations, jurisdictions, and time. Human review is required for consequential use.

Citation

@inproceedings{mohseni2026legal,
  title         = {Legal Guidance in the Wild: How Users Seek Legal Help in Real-World {LLM} Conversations},
  author        = {Mohseni, Amirreza and Spanakis, Gerasimos},
  booktitle     = {Legal Knowledge and Information Systems (JURIX 2026)},
  year          = {2026},
  eprint        = {2610.25159},
  archivePrefix = {arXiv}
}