CaseLawModernBERT-base
CaseLawModernBERT-base is a legal domain-adapted version of ModernBERT-base, further pre-trained on US court opinions using the masked language modeling objective. It supports sequences up to 8,192 tokens, enabling full-length court opinion processing.
Model Description
This model was obtained by further pre-training the ModernBERT-base checkpoint on 8.3 million US court opinions (13 billion words) from the Collaborative Open Law Data (COLD). Pre-training used a masked ratio of 30%, a learning rate of 3e-4 with a linear scheduler, and weight decay of 8e-5, with a batch size of 512 and a maximum sequence length of 8,192 tokens.
The model architecture and training setup are analogous to ModernBERT, incorporating flash attention, ROPE embeddings, and alternating local/global attention layers. To run the model, install the environment described in ModernBERT.
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "ai-law-society-lab/CaseLawModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
Performance
Results on legal NLP benchmarks (test set, average over 5 runs):
| Model | LexGLUE/SCOTUS (Micro-F1) | LexGLUE/CaseHold (Micro-F1) | LePaRD (rc@10) | BarExam QA (rc@10) |
|---|---|---|---|---|
| ModernBERT-base | 65.3 | 74.8 | 70.6 | 2.6 |
| CaseLawModernBERT-base | 67.5 | 76.1 | 71.5 | 5.3 |
See the paper for full results including confidence intervals and additional benchmarks.
Intended Use
This model is intended for NLP tasks related to US court opinions, including:
- Text classification (e.g., topic classification of court opinions)
- Multiple-choice question answering over legal text
- Passage retrieval and reranking in legal search systems
- Computing embeddings of legal passages
Note: We caution users to carefully validate this model before deployment. Legal AI applications can have serious consequences, and model biases may cause systematically worse performance for certain areas of law or categories of litigants.
Limitations
- Pre-trained only on US court opinions; performance on non-US legal text (e.g., European Court of Human Rights) may be lower than the base ModernBERT checkpoint.
- Evaluated primarily on LexGLUE and LePaRD tasks; performance on other legal NLP tasks (e.g., judicial outcome prediction) has not been assessed.
- The training corpus (court opinions converted from PDF) may contain OCR and formatting noise.
Citation
TODO
License
Apache 2.0, consistent with the base ModernBERT license.