sot-phase1-wikitext103-discovered_retrained-seed123-tc017

Part of the Self-Organising Transformer research project: phase1, dataset wikitext103, condition discovered_retrained, seed 123.

The structure the gated supernet selected, trained from scratch with fresh weights.

Read this first

• On WikiText-103 the retrained structure loses to the 8-layer and half-FFN dense baselines.

Full context: RESEARCH_LOG.md, the 2026-10-01 entry.

Intended use and limitations

• Intended use: reproduce and inspect the results of the Self-Organising Transformer study.
• Not for: production or downstream use. This is a small research model trained on WikiText-103 only. It is not instruction-tuned or safety-tuned. Its zero-shot benchmark scores are close to chance.
• It needs the project code and transformers<5.0 (see How to use).

How to use

The model class is defined in the project repository and is not registered with AutoModel, so the Hub widget and AutoModelForCausalLM do not work. Use the repository code (it needs transformers<5.0; 5.x corrupts the reload of this model):

git clone https://github.com/sbartlett97/self-organising-transformer
cd self-organising-transformer && pip install -r requirements.txt
python generate.py sbartlett97/sot-phase1-wikitext103-discovered_retrained-seed123-tc017 --prompt "The history of the city begins in"

from ablation_importance import resolve_checkpoint
from evaluate import load_model
from transformers import AutoTokenizer

path = resolve_checkpoint("sbartlett97/sot-phase1-wikitext103-discovered_retrained-seed123-tc017")
model = load_model(str(path), None).eval()
tok = AutoTokenizer.from_pretrained(path)
logits = model(tok("The history of the city begins in", return_tensors="pt").input_ids)["logits"]

Performance

evaluate.py perplexity: 100 val batches of 8 x 256 tokens.

Perplexity: 33.15

• loses to the matched dense baseline (28.11) by 17.9%

Training trajectory

No training metrics recorded.

Hyperparameters