NagaYu/flashback-first-bad-step

Flashback: first-bad-step detector

Given a run's sketch index -- a few hundred bytes of summary statistics per training step -- this model reports the step at which the run first went wrong.

It is small on purpose: 10 logistic weights plus 9 fitted consensus hyperparameters. The interesting part is not the model, it is that the input is cheap enough to record for every step of every run.

Results on held-out runs

detector	mean abs. error (steps)	exact	within 2	never detected
loss curve alone	0.50	2/6	4/6	2
this model	0.00	6/6	6/6	0

Train split: 12 runs, MAE 0.00. Splits are by run: no step of a test run appears in training.

Usage

from flashback.detector import FirstBadStepDetector
from flashback.sketch import StepSketchIndex
from huggingface_hub import snapshot_download

det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step"))
report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r"))
print(report.first_bad_step, report.confidence, report.top_metrics)

To produce the sketch index in the first place, add three lines to your training loop:

from flashback.integrations import FlashbackCallback
trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])

Fitted configuration

{
  "win": 32,
  "gap": 1,
  "mode": "diff",
  "k_cal": 8.0,
  "k_min": 8.0,
  "min_votes": 3,
  "vote_window": 2,
  "exclude": [
    "lr",
    "step_wall_ms",
    "batch_id_mean",
    "batch_uniq_frac"
  ],
  "use_logistic": true,
  "logistic_threshold": 0.5,
  "version": "1"
}

Feature weights

feature	weight
bias	-9.8951
log1p_max_z	+0.1471
log1p_top5_mean_z	+0.1950
frac_metrics_firing	+18.4084
hard_signal	+1.2226
log1p_z_grad_norm	+0.3008
log1p_z_update_max	+0.1584
log1p_z_loss	+0.0378
log1p_z_hist_tail	+0.2497
rel_position	+15.0875

Limitations

• It needs a healthy stretch of the run to calibrate against; a run that is broken from step 0 has no baseline and will not be localised.
• It reports regime changes, so a fault that ramps in smoothly over hundreds of steps has no single correct answer and the reported step will be early or late by roughly the ramp length.
• It was fitted on small Transformers (a few hundred thousand to a few million parameters). The features are scale-free by construction, but transfer to very large models is not demonstrated here.
• No claim of bitwise determinism is made anywhere in Flashback; this is a statistical detector over recorded statistics.

Generated by Flashback v0.1.0.