Pruned TCFD disclosure classifier built by weight editing (no gradient descent)

A smaller RobertaForSequenceClassification derived from climatebert/distilroberta-base-climate-tcfd (commit 970630beedc2), ClimateBERT's classifier of climate-related disclosure paragraphs into TCFD recommendation categories. An editor LLM chose which encoder layers and feed-forward units to keep and re-fit the remaining weights in closed form from a budgeted number of forward passes on the model's public training split; no backprop, no optimizers, no extra data.

Labels: governance, risk, strategy, metrics

edit_manifest.json, structure.json and param_count.json in this folder record the exact diff against the starting checkpoint, the pruning structure and the parameter accounting.

from transformers import pipeline
clf = pipeline("text-classification", model="sidman007/wsp-climatebert-tcfd-r1-p50")
clf("The board's risk committee reviews climate-related risks twice a year.")

Training data of the start model and practice data of the editor: climatebert/tcfd_recommendations (CC BY-NC-SA 4.0, research use). The model was never trained on the test split used for the accuracy line below.

Held-out test accuracy: 78.75% · macro-F1: 0.7206 · target (accuracy >= 76.50%, macro-F1 >= 0.6896): pass

Run: r1/p50 (50 forward passes, 32 used, $10.14 of editor API spend) · CPU latency 17.3 ms/paragraph at batch 1 and 19.6 ms at batch 16 (4 threads, 512-token window), against 25.2 / 30.1 ms for the published model.

Source, logs and protocol: https://github.com/siddharth-ghatti/weight-space-programming-poc

How it was built

The editor kept all six encoder layers and narrowed every feed-forward block from 3,072 to 1,536 units. For each layer it read the block's actual inputs on practice paragraphs (token-level for the early layers, the <s> position only for layer 5, since that is all the classification head reads), kept the units with the largest output-weight energy times activation variance, and refit the block's output.dense.weight and bias by ridge regression so the kept units reproduce the full block's output, dropped units included. Attention, LayerNorms, input projections and the classification head are the published model's. Every fit is a closed-form solve on activations returned by forward passes; nothing was trained. The checkpoint rebuilds bit-for-bit from the pinned published checkpoint plus the recorded diff (scripts/reins/apply_edit.py --check in the source repository).

Caveats

• Research use only. The published model's training data and the editor's practice data, climatebert/tcfd_recommendations, are licensed CC BY-NC-SA 4.0 (non-commercial). This pruned model was refit on that data, so treat it as a research artifact; commercial deployment needs the dataset authors' permission or a refit on licensed data.
• Small test set. The accuracy line comes from 320 held-out paragraphs (197 strategy, 49 metrics, 48 risk, 26 governance); one paragraph is 0.31 accuracy points and one governance paragraph moves its recall by 3.8 points. The safe reading is that this model is not worse than the published one, not that it is better. One seed.
• The published model's own limits carry over. It scores 77.5% on this test set and is trained on paragraphs, not sentences; its probability outputs were not calibrated by us.
• Not production-ready. See the project README and docs/executive-summary.md in the source repository for the calibration, real-data, robustness and governance work that would be needed.