stationhouse
sae-64k / sae-128k probes (2026-11-06)
Width probes for the Marin-32B neuron-atlas SAE lane: layers 45/57/63 at width 65536 (sae-64k/) and 131072 (sae-128k/), trained from the shared sbg5 A100 lane. Each layer dir holds final + plateau checkpoints, census.parquet, stats.json, train/val logs, sha256.txt (mid-step checkpoints excluded per retention policy). Width verdict: 64k adopted for the full sweep; 128k stopped (median val gain 0.0044 < 0.02). sae-64k/layer=NN also carries the ongoing full-sweep finals.
sae-64k — full 64-layer sweep (canonical)
Marin-32B neuron-atlas TopK SAE, width 65536, k=32, all 64 layers, trained on the sbg5 A100 lane (job 35453930, 2026-11-06/07; ~44-75 min/layer, plateau-early stopping, best-val checkpoints). Per layer: sae_LNNw64_final_s* (canonical finals; the byte-identical plateau copies were sha256-verified identical and culled before upload), stats.json, val_log.jsonl (val-split holdout, trailing 1.92M rows), train_log.jsonl, sha256.txt, census_uniform.parquet + stats_uniform.json (uniform row-group spread, fvu labeled fvu_atlas_uniform) and census_prefix.parquet (original corpus-prefix census, retained for provenance). fvus from different distributions are never compared directly — see the standing methodology note in the ops_sh pad. Layers 45/57/63 dirs also hold the earlier width-probe files (feature-preview-64k.); the sweep finals are canonical. grid_64k_.png at sae-64k/ root.
INPUT-CONVENTION CAVEAT (2026-11-07) - READ BEFORE COMPARING FVUS
The sae-64k sweep was trained AND validated on RAW activations: a layer-id parse failure in the training launch (the 'LNNw64' layer-label failed the label-parse) silently set the per-neuron scale vector to None. The sae-32k family was trained WITH per-neuron scale normalization (plain LNN labels parsed correctly). Consequences:
- 32k and 64k checkpoints expect DIFFERENT input conventions.
- Cross-width fvu comparisons are INVALID until conventions are unified.
- Within a width family, val-vs-val and census-vs-census remain valid.
- sae-64k census files: census_uniform.parquet / stats_uniform.json are the raw-acts v2 redo (input_convention: raw_acts, fvu labeled fvu_atlas_uniform); the scaled-input run is preserved as _scaled_ARTIFACT. (its ~1.0 fvus measured the normalization mismatch, not model quality). census_prefix.parquet is the original corpus-prefix census (also a scaled-input artifact).
- trainer val_fvu (trailing-holdout, raw acts) remains the reference number.