sot-phase1-tinystories-sot_compact_gatefix-seed123

Part of the Self-Organising Transformer research project: phase1, dataset tinystories, condition sot_compact_gatefix, seed 123.

Physically smaller model built from the gated supernet's selected units, with the inherited weights (gate magnitudes folded in) and no further training.

Read this first

• Perplexity corrected on 2026-10-06: the first values came from a transformers==5.17.0 install that corrupts the reload of this model. The lm-eval scores were not re-run after that fix. At this model size they are at chance level in any case.
• With inherited weights and no further training, this model collapses. Selection by gate value dropped layer 0, the most important layer (RESEARCH_LOG.md, 2026-10-09). With layers 0-1 kept and a 300-step fine-tune, most of the loss is recovered.

Full context: RESEARCH_LOG.md, the 2026-09-29 entries (runs) and the 2026-10-06 entry (corrected perplexity).

Intended use and limitations

• Intended use: reproduce and inspect the results of the Self-Organising Transformer study.
• Not for: production or downstream use. This is a small research model trained on TinyStories only. It is not instruction-tuned or safety-tuned. Its zero-shot benchmark scores are close to chance.
• It needs the project code and transformers<5.0 (see How to use).

How to use

The model class is defined in the project repository and is not registered with AutoModel, so the Hub widget and AutoModelForCausalLM do not work. Use the repository code (it needs transformers<5.0; 5.x corrupts the reload of this model):

git clone https://github.com/sbartlett97/self-organising-transformer
cd self-organising-transformer && pip install -r requirements.txt
python generate.py sbartlett97/sot-phase1-tinystories-sot_compact_gatefix-seed123 --prompt "Once upon a time"

from ablation_importance import resolve_checkpoint
from evaluate import load_model
from transformers import AutoTokenizer

path = resolve_checkpoint("sbartlett97/sot-phase1-tinystories-sot_compact_gatefix-seed123")
model = load_model(str(path), None).eval()
tok = AutoTokenizer.from_pretrained(path)
logits = model(tok("Once upon a time", return_tensors="pt").input_ids)["logits"]

Performance

evaluate.py perplexity: 100 val batches of 8 x 256 tokens.

Perplexity: 525.03 (corrected 2026-10-06)

• loses to the matched dense baseline (4.61) by 11288.9%

Gates of the source supernet

Threshold (auto-median of this checkpoint's layer gates): 0.2315

Training trajectory

No training metrics recorded.

Hyperparameters