ModernBERT-large for primary legal-topic classification

Paper: Legal Guidance in the Wild | Code: GitHub

This checkpoint assigns a primary legal topic to conversations in which the user is seeking legal guidance. It is the second stage of the full-conversation ModernBERT-large cascade from the Legal Guidance in the Wild study. Input contains chronological user messages only; assistant messages are excluded. The first-stage detector is modernbert-large-v3-seeks-guidance-user-len4096-seed42.

This is a research classifier, not a legal-advice system. Its topic prediction must not be treated as a determination of jurisdiction, rights, or legal merit.

Labels

The 14 labels are: FAMILY_AND_ESTATES, HOUSING_AND_PROPERTY, EMPLOYMENT_AND_LABOR, IMMIGRATION_AND_CITIZENSHIP, CRIMINAL_LAW, TAX_LAW, CONSUMER_AND_PERSONAL_FINANCE, BUSINESS_AND_COMMERCIAL, TORTS_AND_CIVIL_LIABILITY, GOVERNMENT_AND_ADMINISTRATIVE, CIVIL_RIGHTS_AND_CONSTITUTIONAL, DATA_PRIVACY_AND_TECHNOLOGY, INTELLECTUAL_PROPERTY, and OTHER. The exact ID mapping is stored in config.json.

Data

Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
Requested revision: main (latest at run time)
Train fingerprint: 6ba4c2696e889276
Validation fingerprint: 403fe118d76c8360
Topic-stage train / validation rows: 750 / 134 guidance-positive conversations
Input mode: chronological user messages only

Dataset access follows the linked repository's sharing settings. The fingerprints identify the exact loaded splits even if main later changes.

Training configuration

Setting	Value
Base model	`answerdotai/ModernBERT-large`
Maximum length	4096
Epochs	10
Learning rate	6e-05
Effective batch size	32
Weight decay	0.01
Class weighting	Yes
Seed	42
Hardware	NVIDIA A100-SXM4-40GB

The checkpoint was selected by silver-validation macro-F1.

Silver-validation results

Evaluation	Accuracy	Macro-F1
Conditional topic stage (N=134)	0.716	0.574
Full large cascade (15-way, N=290)	0.776	0.536

The paired first-stage guidance detector obtains 0.893 macro-F1 and 0.887 positive-class F1. Gold-set results are reported below.

Gold-set results

Scored once on the 200 human-adjudicated gold conversations after all modeling decisions were frozen (paper, Tables 4 and 6; per-example predictions are in the GitHub repository).

Evaluation	Accuracy	Macro-F1
Guidance (N=200)	0.845	0.845
Topic, gold guidance-positive (N=84)	0.762	0.671
Full large cascade (N=200)	0.755	0.593

Inference

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "AmirMohseni/modernbert-large-v3-primary-topic-user-len4096-seed42"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

# Run only after the paired guidance detector predicts guidance-seeking.
user_only_conversation = "\n".join(
    f"User: {m['content']}" for m in conversation if m["role"] == "user"
)
inputs = tokenizer(user_only_conversation, return_tensors="pt", truncation=True,
                   max_length=4096)
topic_id = int(model(**inputs).logits.argmax(-1)[0])
topic = model.config.id2label[topic_id]

Limitations

The model was trained on English-language public LLM interaction logs with silver labels, one source, and one seed. Several topics have limited validation support, so macro-F1 is unstable and topic-specific errors can be substantial. Inputs beyond 4,096 tokens are truncated. The taxonomy is jurisdiction-agnostic and cannot substitute for legal triage by a qualified professional. Human review is required for consequential use.

Citation

@inproceedings{mohseni2026legal,
  title         = {Legal Guidance in the Wild: How Users Seek Legal Help in Real-World {LLM} Conversations},
  author        = {Mohseni, Amirreza and Spanakis, Gerasimos},
  booktitle     = {Legal Knowledge and Information Systems (JURIX 2026)},
  year          = {2026},
  eprint        = {2610.25159},
  archivePrefix = {arXiv}
}

AmirMohseni/modernbert-large-v3-primary-topic-user-len4096-seed42