This release accompanies work focusing on speeding up stage 1 of TabICLv2 pretraining.
This stage-1 trainer runs at 1.27 s/step vs 3.8 s/step for the official tabiclv2 trainer on single H100, completing the 500K-step stage-1 recipe in ~7 days instead of ~22 H100-days. Prior generation on a separate CPU with a deterministic, replayable stream (bit-exact across restarts) to avoid CPU bottlenecks.
Stages 2 and 3 were then run to completion so the stage-1 result can be judged by a full, evaluable model.
| stage | recipe | wall time (1× H100) | official (same hw, measured/est.) |
|---|---|---|---|
| 1 | 500K steps, 1,024 rows/dataset, LR 8e-4 | ~7 days (1.27 s/step) | ~22 days (3.8 s/step, measured) |
| 2 | 40K steps, 400–10,240 rows log-uniform, LR 1e-4 | ~2 days | — |
| 3 | 10K steps, 400–60,000 rows log-uniform, LR 2e-5 | ~2.3 days | — |
Classification only (max_classes=10); the regressor was not trained.
| benchmark | ours | reference (released v2) |
|---|---|---|
| TabArena-Lite | Elo 1537.6 | Elo 1557.7 |
| TALENT (181 clf datasets), mean acc | 0.8400 (W/T/L 50/30/101) | 0.8426 |
| Large-dataset suite (15–48K rows), mean acc | 0.9133 | 0.9204 |
| file | stage | steps | note |
|---|---|---|---|
| stage1/step-{50000..500000}.ckpt | 1 | 50K–500K | 6 snapshots |
| stage2/step-{10000..40000}.ckpt | 2 | 10K–40K | 4 snapshots |
| stage3/step-{2500..10000}.ckpt | 3 | 2.5K–10K | stage3/step-10000.ckpt = final model |
Research release from Nolano AI (Apache-2.0). Training/evaluation code and replication checkpoints by Ayush Kaushal, accompanying a stage-1 training-efficiency study. These are research artifacts, not a supported product and no maintenance or support is implied.
stage1/step-100000.ckpt 110 MBstage1/step-200000.ckpt 110 MBstage1/step-300000.ckpt 110 MBstage1/step-400000.ckpt 110 MBstage1/step-500000.ckpt 110 MBstage1/step-50000.ckpt 110 MBstage2/step-10000.ckpt 110 MBstage2/step-20000.ckpt 110 MBstage2/step-30000.ckpt 110 MBstage2/step-40000.ckpt 110 MBstage3/step-10000.ckpt 110 MBstage3/step-2500.ckpt 110 MBstage3/step-5000.ckpt 110 MBstage3/step-7500.ckpt 110 MB