Flashback: first-bad-step detector
Given a run's sketch index -- a few hundred bytes of summary statistics per training step -- this model reports the step at which the run first went wrong.
It is small on purpose: 10 logistic weights plus 9 fitted consensus hyperparameters. The interesting part is not the model, it is that the input is cheap enough to record for every step of every run.
Results on held-out runs
| detector | mean abs. error (steps) | exact | within 2 | never detected |
|---|---|---|---|---|
| loss curve alone | 0.50 | 2/6 | 4/6 | 2 |
| this model | 0.00 | 6/6 | 6/6 | 0 |
Train split: 12 runs, MAE 0.00. Splits are by run: no step of a test run appears in training.
Usage
from flashback.detector import FirstBadStepDetector
from flashback.sketch import StepSketchIndex
from huggingface_hub import snapshot_download
det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step"))
report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r"))
print(report.first_bad_step, report.confidence, report.top_metrics)
To produce the sketch index in the first place, add three lines to your training loop:
from flashback.integrations import FlashbackCallback
trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])
Fitted configuration
{
"win": 32,
"gap": 1,
"mode": "diff",
"k_cal": 8.0,
"k_min": 8.0,
"min_votes": 3,
"vote_window": 2,
"exclude": [
"lr",
"step_wall_ms",
"batch_id_mean",
"batch_uniq_frac"
],
"use_logistic": true,
"logistic_threshold": 0.5,
"version": "1"
}
Feature weights
| feature | weight |
|---|---|
bias |
-9.8951 |
log1p_max_z |
+0.1471 |
log1p_top5_mean_z |
+0.1950 |
frac_metrics_firing |
+18.4084 |
hard_signal |
+1.2226 |
log1p_z_grad_norm |
+0.3008 |
log1p_z_update_max |
+0.1584 |
log1p_z_loss |
+0.0378 |
log1p_z_hist_tail |
+0.2497 |
rel_position |
+15.0875 |
Limitations
- It needs a healthy stretch of the run to calibrate against; a run that is broken from step 0 has no baseline and will not be localised.
- It reports regime changes, so a fault that ramps in smoothly over hundreds of steps has no single correct answer and the reported step will be early or late by roughly the ramp length.
- It was fitted on small Transformers (a few hundred thousand to a few million parameters). The features are scale-free by construction, but transfer to very large models is not demonstrated here.
- No claim of bitwise determinism is made anywhere in Flashback; this is a statistical detector over recorded statistics.
Generated by Flashback v0.1.0.