ModernBERT-large for full-conversation legal-guidance detection

Paper: Legal Guidance in the Wild | Code: GitHub

This checkpoint predicts whether a conversation's user is seeking legal guidance. It is the first stage of the full-conversation ModernBERT-large cascade from the Legal Guidance in the Wild study. Input contains chronological user messages only; assistant messages are excluded. The paired topic model is modernbert-large-v3-primary-topic-user-len4096-seed42.

This is a research routing model, not a legal-advice system. It must not be used to decide whether a person has a valid claim or needs professional counsel.

Labels

False: no legal guidance sought
True: legal guidance sought

Data

Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
Requested revision: main (latest at run time)
Train fingerprint: 6ba4c2696e889276
Validation fingerprint: 403fe118d76c8360
Train / validation conversations: 1,632 / 290
Input mode: chronological user messages only

Dataset access follows the linked repository's sharing settings. The fingerprints identify the exact loaded splits even if main later changes.

Training configuration

Setting	Value
Base model	`answerdotai/ModernBERT-large`
Maximum length	4096
Epochs	3
Learning rate	5e-05
Effective batch size	32
Weight decay	0.01
Seed	42
Hardware	NVIDIA A100-SXM4-40GB

The checkpoint was selected by validation macro-F1, followed by guidance threshold selection. The latest complete run selected threshold 0.38.

Silver-validation results

Evaluation	Macro-F1	Positive F1	Accuracy
Guidance stage (N=290)	0.893	0.887	--
Full large cascade (15-way, N=290)	0.536	--	0.776

The paired topic checkpoint obtains 0.716 accuracy and 0.574 macro-F1 on the 134 silver guidance-positive validation conversations. Gold-set results are reported below.

Gold-set results

Scored once on the 200 human-adjudicated gold conversations after all modeling decisions were frozen (paper, Tables 4 and 6; per-example predictions are in the GitHub repository).

Evaluation	Accuracy	Macro-F1
Guidance (N=200)	0.845	0.845
Topic, gold guidance-positive (N=84)	0.762	0.671
Full large cascade (N=200)	0.755	0.593

Inference

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "AmirMohseni/modernbert-large-v3-seeks-guidance-user-len4096-seed42"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

# conversation: list of {"role", "content"} messages; only user turns are used.
user_only_conversation = "\n".join(
    f"User: {m['content']}" for m in conversation if m["role"] == "user"
)
inputs = tokenizer(user_only_conversation, return_tensors="pt", truncation=True,
                   max_length=4096)
logits = model(**inputs).logits
probability = logits.softmax(-1)[0, model.config.label2id["True"]].item()
seeks_guidance = probability >= 0.38

The threshold 0.38 is the frozen validation-selected operating point and is the value used for every reported result. The repository's class labels alone do not encode it, so an argmax over the two logits will not reproduce the cascade predictions.

Limitations

The checkpoint was trained on English-language public LLM interaction logs with silver labels, one source, and one seed. Inputs beyond 4,096 tokens are truncated. The dataset is jurisdiction-agnostic, may contain sensitive content, and is not representative of all legal-help seekers. Calibration and accuracy may shift across platforms, populations, jurisdictions, and time. Human review is required for consequential use.

Citation

@inproceedings{mohseni2026legal,
  title         = {Legal Guidance in the Wild: How Users Seek Legal Help in Real-World {LLM} Conversations},
  author        = {Mohseni, Amirreza and Spanakis, Gerasimos},
  booktitle     = {Legal Knowledge and Information Systems (JURIX 2026)},
  year          = {2026},
  eprint        = {2610.25159},
  archivePrefix = {arXiv}
}

AmirMohseni/modernbert-large-v3-seeks-guidance-user-len4096-seed42