FedProc-180M v0

Size note. The checkpoint is ~149M parameters (ModernBERT-base encoder plus four task heads). "180M" in the name is a misnomer kept so existing links keep working.

A compact, multi-task transformer for federal procurement text. One shared ModernBERT-base encoder, four task heads. Trained for the open FedProc-Bench.

Important framing — this is a research model for NLP classification and extraction tasks on federal contracting text. It does not determine bid eligibility, compliance, legal status, or any government-recognized designation. Not affiliated with the U.S. government, GSA, SAM.gov, or acquisition.gov.

What it does

Four heads share a single ModernBERT-base backbone:

The headline task is task 4. Frontier LLMs are known to invent FAR clause numbers when extracting citations from procurement text. This model is specialized for that task.

Headline results (v0 test split, 243 records)

Test split is 155 FAR clauses + 23 real SAM solicitations + 65 Claude-generated synthetic excerpts. The honest read is on the real FAR slice, which is not in any frontier model's family distribution:

On the headline metric, FedProc-180M matches Haiku's F1 while inventing fewer than half as many fake clause numbers. GPT-4o is stronger on both measures; latency and per-call cost were not measured for this table.

Aggregated overall scores (mean of per-task macro-F1; task 4 is entity F1):

Why the aggregate looks low: tasks 2 and 3 only see real solicitation titles in v0 (no description text), because the SAM.gov API daily quota blocked the description fetch during the v0 run. Task 4, which uses real procurement text from synthetic and FAR sources, is the head where this model is competitive. v0.1 with full SAM description text should move tasks 1, 2, 3 substantially.

How to use

import json, torch
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download

# Pull the model package (model.pt + tokenizer + thresholds + this README)
local = snapshot_download("raihan-js/fedproc-180m-v0")

# Reconstruct the multi-task model class. NOTE: that model code is not published yet
# (https://huggingface.co/datasets/raihan-js/fedproc-bench holds the benchmark only).
# The weights are the PyTorch state dict in model.pt (~149M parameters).
from models.multi_task.model import FedProcModel
from models.multi_task.labels import (
    TASK1_ID_TO_LABEL, TASK2_ID_TO_LABEL, TASK3_LABELS, N_TASK3,
)

tok = AutoTokenizer.from_pretrained(local)
model = FedProcModel(encoder_name="answerdotai/ModernBERT-base")
model.load_state_dict(torch.load(f"{local}/model.pt", map_location="cpu", weights_only=True))
model.eval()

thresholds = json.loads(open(f"{local}/task3_thresholds.json").read())["thresholds"]

text = "This Solicitation is set aside for HUBZone small business concerns. " \
       "Clauses include 52.219-3, 52.204-7, and DFARS 252.225-7042."

enc = tok(text, return_tensors="pt", return_offsets_mapping=True, truncation=True, max_length=512)
with torch.no_grad():
    out = model(input_ids=enc["input_ids"], attention_mask=enc["attention_mask"])

# Task 1 / 2: argmax over softmax logits
notice = TASK1_ID_TO_LABEL[out.task1_logits.argmax(-1).item()]
sector = TASK2_ID_TO_LABEL[out.task2_logits.argmax(-1).item()]
# Task 3: per-class threshold
probs3 = torch.sigmoid(out.task3_logits)[0].tolist()
setasides = [TASK3_LABELS[i] for i in range(N_TASK3) if probs3[i] >= thresholds[TASK3_LABELS[i]]]
# Task 4: decode BIO spans into clause numbers using offset_mapping
# (see models/multi_task/_decode_bio in benchmark/run_baselines.py)

Training data

• 150 real SAM.gov solicitations (metadata only — descriptions blocked by daily quota during v0)
• 1,032 FAR and DFARS clauses pulled from the ECFR API (acquisition.gov data, public domain)
• 433 Claude-generated synthetic solicitation excerpts that cite real cached clauses, used both for class balance (rare set-asides) and to provide task-4 training text. Validation rejected ~13% of generations that didn't cite their pinned clauses.

Train / val / test split: 1129 / 243 / 243 records. Stratified per source. See the dataset card for full provenance and label maps.

Training setup

• Encoder: answerdotai/ModernBERT-base (149M parameters)
• Batch size 16, max sequence 512 tokens
• AdamW, lr 5e-5, weight decay 0.01, 6% linear warmup
• Mixed precision (AMP), 6 epochs
• Task loss weights: 1 / 1 / 1 / 1
• Single RTX 3060 (12GB), ~4.3 minutes wall time
• Code: models/multi_task/ in the FedProc-Bench repo

Calibration

Task 3 (multi-label set-aside) needed per-class threshold calibration. The default 0.5 produces all-zero predictions because sigmoid outputs saturate in the 0.0–0.4 range. Calibrated thresholds (Youden-style F1-max on val) live in task3_thresholds.json in this repo. They lift task-3 micro-F1 from 0.000 to 0.293 on val without any retraining.

Known limitations

• v0 was trained on only 1,129 records because the SAM.gov daily quota blocked the planned 10k-record fetch. v0.1 retrain with full data is queued.
• Task 2 (NAICS sector) is essentially fed only solicitation titles in v0 because description text was unavailable. It is the weakest head.
• The model has not been evaluated on solicitations from before its ECFR snapshot date (2026-05-08) for drift. FAR clauses are revised periodically.
• The benchmark's test split contains 65 Claude-generated synthetic records; evaluation on those is biased in favor of Claude-family models. The per-source breakdown in the leaderboard makes this explicit.

License

Apache 2.0.

Citation

@software{fedproc_bench_v0,
  author  = {Raihan Sikder},
  title   = {FedProc-Bench v0: a multi-task benchmark for federal procurement NLP},
  year    = {2026},
  url     = {https://huggingface.co/datasets/raihan-js/fedproc-bench}
}