Pruned insurance sentence classifier built by weight editing (no gradient descent)

A smaller BertForSequenceClassification derived from bert-base-uncased. Starting model: the weight-edited classifier from runs/v1/p50 (sidman007/wsp-bert-insurance-v1-p50, 98.31% held-out accuracy, 109,488,392 parameters). An editor LLM chose which encoder layers and feed-forward units to keep and re-fit the remaining weights in closed form from a budgeted number of forward passes; no backprop, no optimizers, no extra data.

Labels: coverage_grant, exclusion, premium_payment, deductible_limits, claims_procedure, definitions, term_cancellation_renewal, conditions_duties

edit_manifest.json, structure.json and param_count.json in this folder record the exact diff against the starting model, the pruning structure and the parameter accounting.

from transformers import pipeline
clf = pipeline("text-classification", model="sidman007/wsp-bert-insurance-c1-p50")
clf("Flood damage is not covered under this policy.")

Held-out test accuracy: 97.92% · macro-F1: 0.9795 · target 97.8%: pass

Run: c1/p50 (50 forward passes, 47 used, $19.39 of editor API spend) · CPU latency 15.8 ms/sentence at batch 1 and 5.8 ms at batch 32 (4 threads), against 34.8 / 14.8 ms for the starting model.

Source, logs and protocol: https://github.com/siddharth-ghatti/weight-space-programming-poc

How it was built

The editor kept original encoder layers 0, 1, 2, 3, 5, 7, 8 and 11 and the 1,024 highest-output-energy feed-forward units in each, refit each narrowed block by ridge regression to reproduce the original model's next hidden state (bottom up, so each refit saw the already-pruned layers below), refit layers 3 and 5 to stand in for the dropped layers 4 and 6, and rebuilt layer 11 as a mean-pooling layer whose feed-forward block is a kernel classifier over 1,008 anchor sentences from the labeled practice set. Every fit is a closed-form solve on activations returned by forward passes; nothing was trained. The checkpoint rebuilds bit-for-bit from bert-base-uncased plus two recorded diffs (scripts/compress/apply_edit.py --check in the source repository).

Caveats

• Accurate but uncalibrated. Like the starting model, this one predicts the correct label with flat softmax probabilities (logits on the practice set range from about -0.5 to 0.9). Multiplying the logits by any constant changes no prediction. Use the argmax label; fit a temperature on your own held-out data before using the scores as confidences.
• Small margin to the target. The 97.8% accuracy bar was met by 0.12 points on one held-out set of 1,779 synthetic sentences, one seed. The editor's own cross-validated estimate was 98.2 to 98.3%; estimates in this experiment ran 0.3 to 1 point optimistic. Treat 97.9% as a point estimate, not a guarantee.
• Synthetic data. Practice and test sentences were generated for the study across seven lines of business and are largely separable by vocabulary. Accuracy on real insurance documents will be lower for this and for the fine-tuned comparison models alike.
• Interpolating classifier. The final layer stores 1,008 practice sentences as kernel anchors and scores the practice set at 100%. Sentences far from the practice distribution may be handled worse than the starting model would.
• Not production-ready. See the project README and docs/executive-summary.md in the source repository for the calibration, real-data, robustness and governance work that would be needed.