ctd-student-e: calibrated typed decisions, target E
A small (about 150M parameters) classifier that reads a text, a question and a list of options in one pass, and returns a calibrated chance for every option: when it says 80%, it should be right about 80% of the time. This repository holds three students trained with different seeds; their chances are averaged.
Use it for a new label set of the same kind: decisions like the training sources, but with categories the student has never seen. It was chosen because on SNIPS (7 unseen intents) it beat every other target on at least one measure, and nothing beat it. It is not reliable for new kinds of judgment: on PubMedQA, the best students in the report were honest about being unsure, but only slightly better than knowing how often each answer occurs.
Details: technical report · blog post · dataset · code. For new label sets, see ctd-student-e; for data like the training sources, see ctd-student-f.
Usage
# pip install "transformers>=4.48" safetensors huggingface_hub
import importlib.util
from huggingface_hub import hf_hub_download
path = hf_hub_download("m-lagnajit/ctd-student-e", "ctd_student.py")
spec = importlib.util.spec_from_file_location("ctd_student", path)
ctd_student = importlib.util.module_from_spec(spec); spec.loader.exec_module(ctd_student)
student = ctd_student.CalibratedStudent.from_pretrained("m-lagnajit/ctd-student-e")
# A choice question: options, ideally with a short description each
student.predict_one(
instructions="Which banking intent best describes this customer message?",
text="I still haven't received my new card, what should I do?",
options=["card arrival", "card delivery estimate", "lost or stolen card", "activate my card"],
descriptions=["asking where an ordered card is", "asking how long delivery takes",
"reporting a lost or stolen card", "asking how to activate a card"])
# A yes/no question: put the candidate label in the question
student.predict_one(
instructions="Is the customer's intent in this message: card arrival (asking where an ordered card is)?",
text="I still haven't received my new card, what should I do?", question_type="yes/no")
predict_one returns a dictionary of chances. For many questions at once, use student.predict(rows) with dicts
holding instructions, state, options, option_descriptions and type ("choice" or "yes/no").
Results
Measured on the files in this repository (three-seed ensemble, each seed with its own dials fitted on the dataset's validation split). Last column: single students, mean (lowest–highest).
| Test | Accuracy | ECE | Log loss | Log loss, single students |
|---|---|---|---|---|
| familiar data (test split) | 0.920 | 0.014 | 0.237 | 0.260 (0.252–0.265) |
| a new label set (SNIPS) | 0.965 | 0.020 | 0.114 | 0.125 (0.122–0.127) |
| a new kind of judgment (PubMedQA) | 0.607 | 0.097 | 0.918 | 0.925 (0.892–0.946) |
For comparison: on PubMedQA, always answering "yes" scores accuracy 0.552, and always predicting the label frequencies scores log loss 0.937. These numbers can differ slightly from the report's ensembles: the report used the original seeds 1 and 2, whose weights were not saved, while this repository contains the same recipe retrained with the same seeds (SNIPS results in the report already used these students).
Training
- Student: ModernBERT-base with one shared linear scorer over a
[MASK]marker before each option. - Target:
corrected_soft_label(the teacher's opinion, with its wrong top picks corrected), on the 16,995 training questions of m-lagnajit/calibrated-typed-decisions. - Recipe: 3 epochs, batch 16, AdamW (weight decay 0.01), learning rates 3e-5 (body) and 1e-3 (scorer), 10% warmup, gradient clipping 1.0, mixed precision; the epoch with the lowest validation log loss is kept.
- Dials: one temperature for choice questions, and one temperature plus a "yes" shift for yes/no questions,
per seed, fitted on validation (
dials.json).
Limitations
- English only; up to 8 options; inputs longer than 512 tokens are cut (only the text, never the options).
- Calibrated on data like its training sources; on unfamiliar kinds of judgment it is honest but knows little.
- Its dials were fitted on familiar data; they may over- or under-correct on very different data.
License and attribution
Weights: Apache 2.0 (as ModernBERT). Trained on data derived from CLINC150 (CC BY 3.0), Banking77 and MASSIVE (CC BY 4.0), GoEmotions, Amazon Polarity and SciCite (Apache 2.0); see the dataset card for citations.
Citation
@techreport{moharana2026ctdreport,
author = {Moharana, Lagnajit},
title = {Calibrated Typed Decisions: What a 7B Teacher's Soft Labels Do, and Don't, Teach a Small Classifier},
institution = {Independent},
type = {Technical report},
year = {2026},
month = nov,
url = {https://lagnajitmoharana.web.app/calibrated-typed-decisions-technical-report.pdf}
}