ModernBERT-large for prefix-aware legal-guidance detection
Paper: Legal Guidance in the Wild | Code: GitHub
This model detects whether legal guidance is already present in the cumulative user-turn prefix of a multi-turn LLM conversation. It was trained for research on early detection and routing of legal information needs. It is not a legal advice system and must not be used to determine whether a person has a valid legal claim.
Task formulation
One example is generated after each user turn. Prefixes before
first_guidance_user_turn_id are negative; the onset prefix and all later
prefixes are positive. Every prefix from a guidance-negative conversation is
negative. The final prefix matches the user-only full-conversation
serialization used in the comparison model.
All prefixes are retained. Each prefix is weighted by the inverse of its conversation's number of user turns, so every conversation has total training weight 1.0.
Data
- Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
- Requested revision:
main(latest at run time) - Resolved revision:
489b87c18e571e8c7e9a3b1c2463b79e50cef120 - Training: 1,632 conversations / 4,865 prefixes
- Validation: 290 conversations / 912 prefixes
- Positive prefixes: 2,042 train / 376 validation
The first-guidance-turn annotations are silver labels and were not independently human-validated for this run. Onset metrics must therefore not be described as gold evaluation.
Training configuration
| Setting | Value |
|---|---|
| Base model | answerdotai/ModernBERT-large |
| Resolved base revision | 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13 |
| GPU | NVIDIA A100-SXM4-40GB (39.5 GiB) |
| Maximum length | 4096 |
| Epochs | 3 |
| Learning rate | 5e-05 |
| Effective batch size | 32 |
| Weight decay | 0.01 |
| Seed | 42 |
| Loss weighting | Equal total weight per conversation |
| W&B run | https://wandb.ai/rl-research-team/legal-guidance-prefix-v3/runs/u0m7x6qb |
Validation results
The threshold 0.36 was selected on the silver validation split by
conversation-weighted prefix macro-F1, with positive-class F1 as tie-breaker.
| Evaluation | Macro-F1 | Positive F1 | AUPRC | Accuracy | Balanced accuracy |
|---|---|---|---|---|---|
| All prefixes, conversation-weighted | 0.8741 | 0.8529 | 0.9181 | 0.8777 | 0.8782 |
| All prefixes, unweighted | 0.8385 | 0.8144 | 0.8973 | 0.8421 | 0.8419 |
| Final/full prefixes | 0.8753 | 0.8667 | 0.9344 | 0.8759 | 0.8757 |
Silver onset analysis
| Measure | Value |
|---|---|
| Negative-conversation false-alarm rate | 0.1410 |
| Pre-onset false-alarm rate | 0.1119 |
| Missed-guidance rate | 0.1045 |
| Exact onset accuracy | 0.7388 |
| Within-one-user-turn accuracy | 0.8060 |
| Mean absolute turn error among detected positives | 0.3500 |
Gold-set results
Scored on the 200 human-adjudicated gold conversations against their conversation-level guidance labels (paper, Tables 4 and 5). Turn 2 covers the 104 conversations with at least two user turns.
| Prefix | Accuracy | Macro-F1 |
|---|---|---|
| After user turn 1 (N=200) | 0.825 | 0.819 |
| After user turn 2 (N=104) | 0.865 | 0.865 |
| After the final user turn (N=200) | 0.860 | 0.859 |
Repository artifacts
prefix_validation_metrics.json: complete metrics and threshold curveprefix_validation_predictions.csv: per-prefix probabilities and predictionsonset_validation_predictions.csv: conversation-level onset predictionsrun_metadata.json: model, data, software, and hardware provenance
Limitations
The dataset is English-language, jurisdiction-agnostic, and derived from public LLM interaction logs that may contain sensitive content. The training and onset labels are silver labels. The model was trained with one seed. Guidance onset was not human-reviewed, so onset metrics are silver-only.
Citation
@inproceedings{mohseni2026legal,
title = {Legal Guidance in the Wild: How Users Seek Legal Help in Real-World {LLM} Conversations},
author = {Mohseni, Amirreza and Spanakis, Gerasimos},
booktitle = {Legal Knowledge and Information Systems (JURIX 2026)},
year = {2026},
eprint = {2610.25159},
archivePrefix = {arXiv}
}