ctd-student-f: calibrated typed decisions, target F
A small (about 150M parameters) classifier that reads a text, a question and a list of options in one pass, and returns a calibrated chance for every option: when it says 80%, it should be right about 80% of the time. This repository holds three students trained with different seeds; their chances are averaged.
Use it for data like the six training sources: short texts and a fixed set of options, such as intents, emotions, review sentiment or citation purposes. It was chosen because it had the best familiar-data accuracy and calibration in the report, and was never reliably beaten there. It is not reliable for new kinds of judgment: on PubMedQA, the best students in the report were honest about being unsure, but only slightly better than knowing how often each answer occurs.
Details: technical report · blog post · dataset · code. For new label sets, see ctd-student-e; for data like the training sources, see ctd-student-f.
Usage
# pip install "transformers>=4.48" safetensors huggingface_hub
import importlib.util
from huggingface_hub import hf_hub_download
path = hf_hub_download("m-lagnajit/ctd-student-f", "ctd_student.py")
spec = importlib.util.spec_from_file_location("ctd_student", path)
ctd_student = importlib.util.module_from_spec(spec); spec.loader.exec_module(ctd_student)
student = ctd_student.CalibratedStudent.from_pretrained("m-lagnajit/ctd-student-f")
# A choice question: options, ideally with a short description each
student.predict_one(
instructions="Which banking intent best describes this customer message?",
text="I still haven't received my new card, what should I do?",
options=["card arrival", "card delivery estimate", "lost or stolen card", "activate my card"],
descriptions=["asking where an ordered card is", "asking how long delivery takes",
"reporting a lost or stolen card", "asking how to activate a card"])
# A yes/no question: put the candidate label in the question
student.predict_one(
instructions="Is the customer's intent in this message: card arrival (asking where an ordered card is)?",
text="I still haven't received my new card, what should I do?", question_type="yes/no")
predict_one returns a dictionary of chances. For many questions at once, use student.predict(rows) with dicts
holding instructions, state, options, option_descriptions and type ("choice" or "yes/no").
Results
Measured on the files in this repository (three-seed ensemble, each seed with its own dials fitted on the dataset's validation split). Last column: single students, mean (lowest–highest).
| Test | Accuracy | ECE | Log loss | Log loss, single students |
|---|---|---|---|---|
| familiar data (test split) | 0.929 | 0.015 | 0.223 | 0.245 (0.239–0.254) |
| a new label set (SNIPS) | 0.958 | 0.020 | 0.126 | 0.140 (0.137–0.143) |
| a new kind of judgment (PubMedQA) | 0.608 | 0.085 | 0.910 | 0.926 (0.875–0.953) |
For comparison: on PubMedQA, always answering "yes" scores accuracy 0.552, and always predicting the label frequencies scores log loss 0.937. These numbers can differ slightly from the report's ensembles: the report used the original seeds 1 and 2, whose weights were not saved, while this repository contains the same recipe retrained with the same seeds (SNIPS results in the report already used these students).
Training
- Student: ModernBERT-base with one shared linear scorer over a
[MASK]marker before each option. - Target:
0.5 × one-hot(gold) + 0.5 × corrected_soft_label, on the 16,995 training questions of m-lagnajit/calibrated-typed-decisions. - Recipe: 3 epochs, batch 16, AdamW (weight decay 0.01), learning rates 3e-5 (body) and 1e-3 (scorer), 10% warmup, gradient clipping 1.0, mixed precision; the epoch with the lowest validation log loss is kept.
- Dials: one temperature for choice questions, and one temperature plus a "yes" shift for yes/no questions,
per seed, fitted on validation (
dials.json).
Limitations
- English only; up to 8 options; inputs longer than 512 tokens are cut (only the text, never the options).
- Calibrated on data like its training sources; on unfamiliar kinds of judgment it is honest but knows little.
- Its dials were fitted on familiar data; they may over- or under-correct on very different data.
License and attribution
Weights: Apache 2.0 (as ModernBERT). Trained on data derived from CLINC150 (CC BY 3.0), Banking77 and MASSIVE (CC BY 4.0), GoEmotions, Amazon Polarity and SciCite (Apache 2.0); see the dataset card for citations.
Citation
@techreport{moharana2026ctdreport,
author = {Moharana, Lagnajit},
title = {Calibrated Typed Decisions: What a 7B Teacher's Soft Labels Do, and Don't, Teach a Small Classifier},
institution = {Independent},
type = {Technical report},
year = {2026},
month = nov,
url = {https://lagnajitmoharana.web.app/calibrated-typed-decisions-technical-report.pdf}
}