legalbert-seeks_guidance

Fine-tuned Legal-BERT classifier that detects whether a user in a multi-turn conversation is seeking legal guidance.

Part of the Legal QA collection.

Model description

Stage 1 of a two-model Legal-BERT routing pipeline:

Legal-BERT is trained with 512-token truncation. Long WildChat threads are truncated from the start, which is a known limitation compared to long-context encoders such as ModernBERT.

Results

Joint pipeline test (with legalbert-primary_topic): legal 84.1%, topic 72.0%, joint 72.0%.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

def serialize(messages, input_mode="user"):
    lines = []
    for msg in messages:
        role = msg["role"]
        if input_mode == "user" and role != "user":
            continue
        lines.append(f"{role.capitalize()}: {msg['content']}")
    return "\n".join(lines)

model_id = "AmirMohseni/legalbert-seeks_guidance"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
input_mode = getattr(model.config, "legal_cls_input_mode", "user")
max_length = getattr(model.config, "legal_cls_max_length", 512)

text = serialize(conversation, input_mode=input_mode)
enc = tokenizer(text, truncation=True, max_length=max_length, return_tensors="pt")
with torch.no_grad():
    pred_id = model(**enc).logits.argmax(dim=-1).item()
print(model.config.id2label[str(pred_id)])

Intended uses & limitations

Use for: encoder baseline comparisons on English legal routing at 512 tokens.

Do not use for: legal advice or deployment without evaluation on your traffic.

Caveats: silver GPT-5.4 labels; severe truncation on long conversations.

Training data

Dataset: AmirMohseni/WildChat-Legal-Classification-V2-Balanced

• Balanced legal / non-legal rows from WildChat-1M with GPT-5.4 structured labels
• Splits: train 1909 · val 106 · test 107

Training procedure

Selected from a 24-trial hyperparameter sweep (LR × input mode × effective batch). See legalbert_classifier_sweep_colab.ipynb.

Sweep note: all top-9 validation runs used user input; best full-conversation run reached only 88.67% F1.

Framework versions

• Transformers 5.9.0 · PyTorch · Datasets