nanodiff-350m-typed-decisions-lam0
Control arm (λ = 0.0) of the typed-decision calibration study: LLaDA-style SFT fine-tune of
Sebasdi/nanodiff-350m-base on pngwn/typed-decisions with the decision-slot CE computed and
logged but not added to the loss (correctness-only training). See the full report.
- 3,000 steps · seed 1337 · batch 32 × seq 512 × grad-accum 2 (32k tok/step) · A100-80GB, ≈ 27.6 min
- Checkpoint:
nanodiff-350m-typed-decisions-lam0.pt(1.5 GB,torch.load→ state dict; load exactly as incode/eval_calibration.py) - Eval artifacts in this repo:
eval_calibration.json,eval_calibration_logits.npz
Eval (test, decision-level; T=1.95 fit on cal)
| group | n | acc | ece_hard | ece_soft | ece_soft|temp | AUROC | acc@.5 |
|---|---|---|---|---|---|---|---|
| ALL | 5214 | 0.667 | 0.333 | 0.065 | 0.041 | 0.922 | 0.957 |
| choice (1-of-10) | 1195 | 0.110 | 0.890 | 0.176 | 0.088 | 0.526 | 0.119 |
| noul | 231 | 0.606 | 0.394 | 0.208 | 0.102 | 0.681 | 0.716 |
| severity (Score) | 338 | 0.704 | 0.296 | 0.053 | 0.081 | 0.838 | 0.947 |
| workflow4 (k=4) | 2436 | 0.852 | 0.148 | 0.023 | 0.060 | 0.880 | 0.998 |
Headline: the masked-slot softmax (ece_soft 0.065) is 5× better calibrated than the
argmax-with-confidence-1.0 reading of the same predictions (ece_hard 0.333), at identical accuracy.
On synthetic severity the posterior is within L1 0.077 of the closed-form Bayes-optimal
posterior (model acc 0.704 vs oracle 0.716; bayes_ece 0.030).
Caveats: the choice domain is at chance (AUROC 0.526) — a knowledge limitation of the 350M
base, not a calibration artifact. escalate/review gold labels disagree with their own Bayes
oracle; see the report.
Scoring (one bidirectional pass)
# Only the answer positions are [MASK]ed. See code/eval_calibration.py (pinned at 65691f24)
# for the exact released path (loading, option-token map, T scaling).
logits = model(x_masked, t=...) # (B, 512, 50304)
probs = logits[:, slot_idx].softmax(-1) # the decision distribution
Sibling arm (λ=1.0): nanodiff-350m-typed-decisions-lam1.