sot-phase1-wikitext103-dense_c425_d768l4-seed42-tc768d013

Part of the Self-Organising Transformer research project: phase1, dataset wikitext103, condition dense_c425_d768l4, seed 42.

Read this first

• This is a dense baseline, not the self-organising method. The gated (SOT) models in this project are compared against it.
• Project result so far: the gated model trails dense by 2-4% perplexity, and no result yet shows a cheaper model at equal quality (README).

Intended use and limitations

• Intended use: reproduce and inspect the results of the Self-Organising Transformer study.
• Not for: production or downstream use. This is a small research model trained for 10,000 steps on WikiText-103 only. It is not instruction-tuned or safety-tuned.
• It needs the project code and transformers<5.0 (see How to use).

How to use

The model class is defined in the project repository and is not registered with AutoModel, so the Hub widget and AutoModelForCausalLM do not work. Use the repository code (it needs transformers<5.0; 5.x corrupts the reload of this model):

git clone https://github.com/sbartlett97/self-organising-transformer
cd self-organising-transformer && pip install -r requirements.txt
python generate.py sbartlett97/sot-phase1-wikitext103-dense_c425_d768l4-seed42-tc768d013 --prompt "The history of the city begins in"

from ablation_importance import resolve_checkpoint
from evaluate import load_model
from transformers import AutoTokenizer

path = resolve_checkpoint("sbartlett97/sot-phase1-wikitext103-dense_c425_d768l4-seed42-tc768d013")
model = load_model(str(path), None).eval()
tok = AutoTokenizer.from_pretrained(path)
logits = model(tok("The history of the city begins in", return_tensors="pt").input_ids)["logits"]

Performance

Not yet evaluated.

Training trajectory

• Final train loss: 3.3773 at step 10000
• Final cost proxy (gate-weighted, not compute): 1.0000
• Final training-time val loss (random val batches): 3.4073 (ppl 30.18) at step 10000

Hyperparameters