Neural-geometry shape fitter: run main_s0
This is a Set Transformer trained only on synthetic shapes. Its input is a 3D point cloud: the top-3 PCA coordinates of a set of activations. It outputs one of 13 shape families and parameters that specify the shape uniquely. The 13 families are blob, clusters, polytope, line, arc/circle, ellipse, helix, PCA-ringing curve, disc, sphere, paraboloid, cone/frustum and torus.
- Code, preregistration and reports: https://github.com/BurnyCoder/geometrical-shape-fitting-in-llm-mechanistic-interpretability
- Paper:
paper.pdfin this repository - Run logs (training, evaluation, activation extraction with the LLM prompts; the counting corpus is logged as the first 200 characters per document, and the exploratory Pythia counting run logs none):
logs/in this repository - Interactive explorer of real and synthetic fits: https://burnycoder.github.io/geometrical-shape-fitting-in-llm-mechanistic-interpretability/
- License: MIT (https://github.com/BurnyCoder/geometrical-shape-fitting-in-llm-mechanistic-interpretability/blob/main/LICENSE)
Results (generated from the results files)
- Synthetic test set (39,000 clouds, 13 families): the method picks the correct family in 87.9% [86.9, 88.9] of clouds; median Chamfer distance to the ground-truth shape 0.156 (RMS-radius units).
- Same 3,120 clouds for every system: method 87.4% [83.7, 90.9], B2 features + boosting 64.5% [59.0, 70.3], B1 classical fit + geometric AIC 52.2% [47.2, 57.5], B1-fixed (post hoc) 54.0% [49.0, 58.7], trivial baselines 7.7-7.9%.
- H1 (macro-F1 method - B1): 0.3741 [0.3410, 0.4095], Holm p < 1e-16: supported.
- H2 (median Chamfer method - B1): -0.0554 [-0.0766, -0.0412], Holm p < 1e-16: supported.
- H5b_refinement (median Chamfer method - no refinement): -0.0033 [-0.0055, -0.0011], Holm p < 1e-16: supported.
- Selection rule: choosing the family by geometric AIC instead of the classifier drops accuracy to 64.6% [63.0, 66.0]. On clouds whose true family is not blob, A5 reaches 70.0% vs the method's 88.1%.
- Geometric AIC cannot select the blob (post hoc, audit 2026-11-14): B1 and the A5 G-AIC rule output blob for 0% and 0% of blob clouds (the method: 86%), a structural property of the noise estimate (proof in reports/deviations.md). On the clouds whose true family is not blob, macro-F1 is 0.880 (method) vs 0.561 (B1), difference 0.319 [0.285, 0.356].
- B1-fixed (post hoc; B1 with per-cloud cluster counts, exact residuals and a blob threshold tau = 0.213 fitted on 780 validation clouds): accuracy 54.0% [49.0, 58.7]; method - B1-fixed macro-F1 0.338 [0.306, 0.375], median Chamfer -0.0608 [-0.0846, -0.0454]. Against B1, the repairs change macro-F1 by +0.036 [0.011, 0.055] and recover 55% of blob clouds.
- Calibration: temperature T = 0.772 (fitted on the validation stream) lowers ECE from 0.044 to 0.022 and NLL from 0.397 to 0.373; the NLL-optimal T on the test set itself is 0.784.
- H3, real activations: the coarse shape class (point set / curve / surface / blob; majority label over random 80% subsamples) agrees with the literature on 7/12 preregistered datasets (exact 95% CI 28-85%), label-permutation p = 0.125: not supported. Post hoc, with the full-cloud label instead of the subsample majority: 8/12. For reference (post hoc): always answering the most frequent expected class, curve, agrees on 8/12; the exact family agrees on 1/12 (full cloud: 2/12).
- H4, controls: matched-covariance Gaussians are labelled blob 86% of the time (mean over all 17 real datasets, exploratory ones included); ordinal-process controls ringing 60% vs helix 40%: supported (mean criterion); 10/12 preregistered datasets reach 80%. Post hoc: smooth Gaussian processes alone are labelled helix in 80%, random walks in 0%.
- Helix rule (post hoc, every full-cloud helix fit with held-out folds): the preregistered two-part rule rejects the helix claim for
C1_gp_0and accepts it forR2_grokking_WE,R5a_llama_weekdays,R5b_llama_months,R5b_llama_months_means, all of which are expected to be non-helix shapes, so the rule is necessary, not sufficient. - Error bars: 95% cluster bootstrap over generator chunks of 25 clouds (audit 2026-11-14), because clouds generated together share N and the R^D-embedding draw.
Results describe the shape of the top-3 PCA projection, not the intrinsic manifold. See the paper for the failure analysis on real activations.
Training summary
{"steps_done": 22643, "minutes": 110.00926675399144, "n_params": 1712093, "resumed_from_step": 10000, "val_acc": 0.90185546875, "val_cd_teacher": 0.2106669193599373}
Usage
from neurogeom.train.loop import load_model
from neurogeom.infer.predict import fit_cloud, describe_fit
model, cfg = load_model("final.pt")
fit, frame, _ = fit_cloud(model, activations, cfg["infer"], temperature=0.7721) # activations: (N, D) numpy array
print(describe_fit(fit, frame))
Author: Libor "Burny" Burian.