slm-125m-base
A 126M-parameter Llama-architecture base model pretrained from scratch on US case law, SEC filings, and educational web text. Trained on 8xH100 for ~28 minutes at a total project cost of roughly $18.
This is a base model. It has had no instruction tuning, no RLHF, and no safety alignment. It completes text; it does not follow instructions.
Architecture
| Parameters | 125,847,552 |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocab | 16384 (byte-level BPE trained on this corpus) |
| Tied embeddings | True |
Training data
1.92B tokens, ~76% legal:
| Source | Tokens | Share |
|---|---|---|
PleIAs/SEC |
819M | 43% |
HFforLegal/case-law |
639M | 33% |
HuggingFaceFW/fineweb-edu |
463M | 24% |
Pipeline: deterministic rule-based cleaning (line filters, boilerplate stripping, repetition and language gates, plus a dictionary-based OCR gate on case law) then MinHash near-dedup, exact dedup, and 13-gram decontamination against the CaseHOLD evaluation set.
Training
2 epochs (3.84B tokens seen), AdamW, cosine schedule to a 6e-05 cooldown, bf16, sequence length 1024, global batch 524,288 tokens.
Results
| Metric | Value |
|---|---|
| Val perplexity (held-out 1%) | 9.07 |
| CaseHOLD (zero-shot, 5-way) | 19.5% |
CaseHOLD is measured on data excluded from training by 13-gram decontamination, so it is not inflated by leakage. Expect near-chance zero-shot performance: a 125M base model with no task tuning is not competitive on multiple-choice legal reasoning, and this number is a baseline for fine-tuning, not a capability claim.
Limitations
Small, base-only, English-only, 1024-token context. Case-law inputs contain OCR noise that survives filtering. Not legal or financial advice and not suitable for any use where an incorrect statement of law carries consequences.