Hierarchical document classifier for the LLM-Mailroom intake pipeline: a
fine-tuned ModernBERT-base encoder with a doc_type head plus one
subclass head per document class. It is the deterministic pre-check in the
BERT-coupled intake overhaul (mailroom-issues #85).
Recommended production revision (M9a primary, Arm B):
m9a-local-20260927-014429
— 3 epochs, loss_lambda_dt=0.65, held-out doc_type accuracy 0.9505,
trainer validation gate_met: true. M9a mailroom-issues #112 subclass
gates remain not met for contract / correspondence; use LLM sorter for
subclass-ambiguous tails.
| Release tag | Arm | Epochs | λ_dt | Held-out doc_type acc | Held-out subclass (cond.) | Trainer gate_met |
#112 gates | Eval on Hub |
|---|---|---|---|---|---|---|---|---|
m9a-local-20260927-014429 |
B (primary) | 3 | 0.65 | 0.9505 | 0.5831 | true | FAIL (2/4) | eval_report_m9a-local-20260927-014429.json |
m9a-local-gpu1-armA-1ep-20260927-025236 |
A (smoke) | 1 | 0.50 | 0.8638 | 0.4839 | false | FAIL (3/4) | eval_report_m9a-local-gpu1-armA-1ep-20260927-025236.json |
Held-out test set: 323 documents, 8,192-token windows, never used for
training, calibration fit, or threshold tuning. Authoritative metrics come from
the eval_report_<tag>.json artifacts (not inline summary.json test fields
when they differ).
Contractor-repo report mirrors (same content as sections below):
M9a-REPORT-m9a-local-20260927-014429,
M9a-REPORT-m9a-local-gpu1-armA-1ep-20260927-025236,
M9a-COMPARE-ArmA-vs-ArmB-20260927.
answerdotai/ModernBERT-base — 22 layers, 768 hidden,
8,192-token context, bf16.doc_type (6 classes) plus 5 subclass
heads (contract, corporate_record, correspondence, insurance_claim,
merger_agreement). MLP heads with dropout 0.1.doc_type
by plurality vote over windows; subclass by plurality over windows whose
doc_type vote is the winning class.temperatures.json).| file | purpose |
|---|---|
model.safetensors |
ModernBERT backbone weights (bf16, ~298 MB) |
heads.pt |
hierarchical head state dicts |
labels.json |
head vocabularies (labels / label2id / id2label / weights) |
temperatures.json |
per-head calibration temperatures |
train_counts.json |
per-(doc_type, subclass) authentic train-row counts (support gate) |
config.json, tokenizer.json, tokenizer_config.json |
backbone config + tokenizer |
summary.json |
full run summary (hyperparameters, per-epoch metrics, selection, test metrics) |
eval_report_<release_tag>.json |
held-out test harness output for that release (on repo root) |
doc_type (6): contract, merger_agreement, corporate_record,
correspondence, insurance_claim, unknown (inference-only abstention —
not a trained class)
contract (24): agency, co_branding, collaboration, consulting, development, distributor, endorsement, franchise, hosting, ip, joint_venture, license, maintenance, manufacturing, marketing, other, outsourcing, promotion, reseller, service, sponsorship, strategic_alliance, supply, transportation
corporate_record (10): articles_of_incorporation, board_resolution, bylaws, charter_amendment, indenture, officer_certificate, other, powers_of_attorney, rights_instrument, subsidiary_list
correspondence (7): demand, email, letter, meeting_request, memo, notice, press_release
insurance_claim (6): auto, carrier, inpatient, outpatient, pde, property
merger_agreement (5): all_cash, all_stock, mixed_cash_stock, mixed_cash_stock_election, other
The backbone is a standard ModernBertModel; the hierarchical heads are a
custom bundle. Load a pinned release tag (never a floating main tip in
production):
from huggingface_hub import snapshot_download
from mailroom_ml.inference import load_bundle, classify_document
model_dir = snapshot_download(
"Lucius-Morningstar/mailroom-modernbert-classifier",
revision="m9a-local-20260927-014429",
)
bundle = load_bundle(model_dir)
result = classify_document(
title="Notice of Default",
text=document_text,
bundle=bundle,
)
# result["doc_type"], result["subclass"], result["confidence"], result["route"]
route == "fast_path" means the calibrated gate passed and the LLM sorter may
be skipped (skip mode + allowlist only); otherwise route the document to the
LLM sorter with the BERT triage as an advisory prior.
contract / correspondence /
corporate_record / merger_agreement on held-out test (M9a Arm B macro-F1
0.02–0.25). Use the doc_type head for routing; route subclass-ambiguous
documents to the LLM sorter. Do not trust skip-mode subclass for these
classes until #112 gates pass.unknown is an inference-only abstention label, not a trained class.Apache-2.0 (inherits answerdotai/ModernBERT-base).
The sections below merge the contractor-repo M9a reports published after local GPU training. Arm B is the production candidate; Arm A is a one-epoch λ ablation smoke on GPU 1.
Run tag: m9a-local-20260927-014429
Run ID: 20260927-064513
Arm: B (primary, GPU 0)
Checkpoint: data/modernbert_training/runs/m9a-local-20260927-014429/latest
Training log: logs/m9a-local-20260927-014429.log
Held-out eval JSON: reports/eval_m9a-local-20260927-014429.json → Hub
eval_report_m9a-local-20260927-014429.json
| Setting | Value |
|---|---|
loss_lambda_dt |
0.65 (Arm B) |
| epochs | 3 |
| batch × grad_accum | 4 × 8 |
| lr | 2e-5 |
| seed | 42 |
| max_length | 8192 |
| label_smoothing | 0.05 |
| weight_mode / cap | inverse / 20 |
| warmup_frac | 0.06 |
| mlp_heads | true |
| freeze_backbone_epochs | 0 |
| model | answerdotai/ModernBERT-base |
| data | Lucius-Morningstar/mailroom-modernbert-training @ prepared stage pin |
| training wall time | ~4.5 h (16,326 s) |
| Epoch | doc_type macro-F1 | doc_type doc acc | subclass objective | selected |
|---|---|---|---|---|
| 1 | 0.9254 | 0.9164 | — | yes (interim) |
| 2 | 0.9345 | 0.9298 | — | yes (interim) |
| 3 | 0.9389 | 0.9331 | 0.3334 | yes (shipped) |
gate_met: true (calibrated doc_type ECE ≤ 0.05, no doc_type macro-F1 regression beyond 0.005) | head | ECE (calibrated) | excluded |
|---|---|---|
| doc_type | 0.02 | no |
| contract | 0.0327 | no |
| insurance_claim | 0.0025 | no |
| corporate_record | 0.071 | yes |
| correspondence | 0.1384 | yes |
| merger_agreement | 0.1083 | yes |
| Metric | Value |
|---|---|
| doc_type_accuracy | 0.9505 (307/323) |
| subclass_accuracy_conditional | 0.5831 (179/307 scorable) |
| window_calibration ECE | 0.0259 |
| window_calibration band ECE | 0.0525 |
| cohort | n_docs | doc_type_accuracy | mean_agreement | window ECE |
|---|---|---|---|---|
| single-window | 274 | 0.9489 | 1.0 | 0.0426 |
| multi-window | 49 | 0.9592 | 0.9724 | 0.0168 |
| head | macro-F1 |
|---|---|
| insurance_claim | 0.9889 |
| corporate_record | 0.2485 |
| merger_agreement | 0.2381 |
| correspondence | 0.1289 |
| contract | 0.0195 |
| Gate | Actual | Threshold | Status |
|---|---|---|---|
| contract test macro-F1 | 0.0195 | ≥ 0.20 | NOT MET |
| correspondence test macro-F1 | 0.1289 | ≥ 0.25 | NOT MET |
| doc_type test accuracy | 0.9505 | ≥ 0.89 | MET |
| window ECE (doc_type calibrated) | 0.0259 | ≤ 0.05 | MET |
Overall #112 gate verdict: FAIL (2/4 gates not met — contract and correspondence subclass heads remain below Verify-grade thresholds; doc_type and calibration gates pass).
gate_met true on validation. Run tag: m9a-local-gpu1-armA-1ep-20260927-025236
Run ID: 20260927-075323
Arm: A (one-time GPU 1 smoke per logs/.m9a-gpu-policy)
Checkpoint: data/modernbert_training/runs/m9a-local-gpu1-armA-1ep-20260927-025236/latest
Training log: logs/m9a-local-gpu1-armA-1ep-20260927-025236.log
Held-out eval JSON: reports/eval_m9a-local-gpu1-armA-1ep-20260927-025236.json → Hub
eval_report_m9a-local-gpu1-armA-1ep-20260927-025236.json
Published with --force-publish (smoke compare; gates not required).
| Setting | Value |
|---|---|
loss_lambda_dt |
0.5 (Arm A) |
| epochs | 1 |
| batch × grad_accum | 4 × 8 |
| lr | 2e-5 |
| seed | 42 |
| max_length | 8192 |
| label_smoothing | 0.05 |
| weight_mode / cap | inverse / 20 |
| warmup_frac | 0.06 |
| mlp_heads | true |
| model | answerdotai/ModernBERT-base |
| data | Lucius-Morningstar/mailroom-modernbert-training @ prepared stage pin |
| training wall time | ~1.55 h (5,579 s) |
| Epoch | doc_type macro-F1 | doc_type doc acc | subclass objective | selection gate_met |
|---|---|---|---|---|
| 1 | 0.8158 | 0.8528 | 0.238 | false |
No checkpoint met the lexicographic selection rule (calibrated doc_type ECE
0.0524 > 0.05 budget on the sole epoch). Trainer reports gate_met
false and selection epoch 0 (placeholder).
| Metric | Value |
|---|---|
| doc_type_accuracy | 0.8638 (279/323) |
| subclass_accuracy_conditional | 0.4839 |
| window_calibration ECE | 0.0397 |
| window_calibration band ECE | 0.0574 |
Note: inline trainer test_metrics.doc_type_acc (0.8669) reflects a
lighter in-train test pass; the authoritative held-out harness is the eval JSON
above.
| cohort | n_docs | doc_type_accuracy | mean_agreement | window ECE |
|---|---|---|---|---|
| single-window | 274 | 0.8723 | 1.0 | 0.1179 |
| multi-window | 49 | 0.8163 | 0.802 | 0.1024 |
Selective-risk sweep refused (no per-head ECE sidecar in artifact for this 1-epoch smoke).
| head | macro-F1 |
|---|---|
| insurance_claim | 0.7552 |
| merger_agreement | 0.16 |
| correspondence | 0.0944 |
| corporate_record | 0.0212 |
| contract | 0.0039 |
| Gate | Actual | Threshold | Status |
|---|---|---|---|
| contract test macro-F1 | 0.0039 | ≥ 0.20 | NOT MET |
| correspondence test macro-F1 | 0.0944 | ≥ 0.25 | NOT MET |
| doc_type test accuracy | 0.8638 | ≥ 0.89 | NOT MET |
| window ECE (doc_type calibrated) | 0.0397 | ≤ 0.05 | MET |
Overall #112 gate verdict: FAIL (expected for 1-epoch λ_dt=0.5 compare arm).
Arm A smoke confirms λ_dt=0.5 + 1 epoch under-trains doc_type vs Arm B: −8.7 pp test doc_type acc and −10 pp conditional subclass acc vs the 3-epoch primary. Use for ablation only; do not promote to production default.
| Arm B (primary) | Arm A (smoke) | |
|---|---|---|
| Run tag | m9a-local-20260927-014429 |
m9a-local-gpu1-armA-1ep-20260927-025236 |
| GPU policy | GPU 0 | one-time GPU 1 exception |
loss_lambda_dt |
0.65 | 0.5 |
| epochs | 3 | 1 |
| Eval JSON (Hub) | eval_report_m9a-local-20260927-014429.json |
eval_report_m9a-local-gpu1-armA-1ep-20260927-025236.json |
| Metric | Arm B | Arm A | Δ (B − A) |
|---|---|---|---|
| doc_type_accuracy | 0.9505 | 0.8638 | +0.0867 |
| subclass_accuracy_conditional | 0.5831 | 0.4839 | +0.0992 |
| window ECE | 0.0259 | 0.0397 | −0.0138 (better B) |
| Arm B | Arm A | |
|---|---|---|
| Selected epoch | 3 | 0 (none qualified) |
Selection gate_met |
true | false |
| Val doc_type macro-F1 | 0.9389 | 0.8158 (e1 only) |
| Val doc_type ECE (cal.) | 0.02 | 0.0524 (e1) |
| Gate | Arm B | Arm A |
|---|---|---|
| contract macro-F1 ≥ 0.20 | NOT MET | NOT MET |
| correspondence macro-F1 ≥ 0.25 | NOT MET | NOT MET |
| doc_type acc ≥ 0.89 | MET | NOT MET |
| window ECE ≤ 0.05 | MET | MET |
| Overall | FAIL (2/4) | FAIL (3/4) |
eval_report_<tag>.json artifacts.| doc_type test acc | source |
|---|---|
| 0.8947 | run-3 (eval_report_run3.json / reports/eval_run3_20260921.json) |
| 0.9505 | Arm B (M9a primary) |
| 0.8638 | Arm A smoke |
Checkpoint runs/20260921-132753 (run_id 20260921-093211). Config: epochs=2,
batch=4, grad-accum=8, lr=2e-5, loss_lambda_dt=0.65, sqrt-inverse weights,
MLP heads. Held-out test doc_type accuracy 0.8947, conditional subclass
0.526, window ECE 0.0203. Superseded for routing by M9a Arm B
(0.9505 doc_type acc).
m9c-20261014-113614 (2026-10-14)Held-out test eval artifact on this repo: eval_report_m9c-20261014-113614.json.
| Gate | Actual | Threshold | Status |
|---|---|---|---|
| contract test macro-F1 | 0.3589 | ≥ 0.2 | MET |
| correspondence test macro-F1 | 0.5249 | ≥ 0.25 | MET |
| doc_type test accuracy | 0.9598 | ≥ 0.89 | MET |
| window ECE (doc_type calibrated) | 0.0189 | ≤ 0.05 | MET |
| head | macro-F1 | support | ECE (calibrated) |
|---|---|---|---|
| contract | 0.3589 | {'affiliate': 2, 'agency': 0, 'co_branding': 5, 'collaboration': 2, 'consulting': 1, 'development': 4, 'distributor': 5, 'endorsement': 0, 'franchise': 1, 'hosting': 2, 'ip': 4, 'joint_venture': 2, 'license': 2, 'maintenance': 4, 'manufacturing': 2, 'marketing': 2, 'other': 0, 'outsourcing': 0, 'promotion': 0, 'reseller': 0, 'service': 8, 'sponsorship': 3, 'strategic_alliance': 4, 'supply': 5, 'transportation': 2} | None |
| corporate_record | 0.7733 | {'articles_of_incorporation': 5, 'board_resolution': 5, 'bylaws': 1, 'charter_amendment': 9, 'indenture': 4, 'officer_certificate': 7, 'other': 0, 'powers_of_attorney': 3, 'rights_instrument': 3, 'subsidiary_list': 10} | None |
| correspondence | 0.5249 | {'demand': 6, 'email': 39, 'letter': 14, 'meeting_request': 5, 'memo': 9, 'notice': 4, 'press_release': 8} | None |
| insurance_claim | 1.0 | {'auto': 34, 'carrier': 20, 'inpatient': 15, 'outpatient': 15, 'pde': 14, 'property': 16} | None |
| merger_agreement | 0.1774 | {'all_cash': 6, 'all_stock': 2, 'mixed_cash_stock': 1, 'mixed_cash_stock_election': 0, 'other': 8} | None |
onnx/model.onnx 611 MBmodel.safetensors 298 MBonnx/model_quantized.onnx 154 MBheads.pt 14 MB