sot-phase2-wikitext103-heuristic-seed42-tc768d013-v9-zeroeffect

Part of the Self-Organising Transformer research project: phase2, dataset wikitext103, condition heuristic, seed 42.

Phase 2: training starts with 33% of units alive; a controller gives birth to dormant units and kills weak ones every 250 steps.

Read this first

• Phase 2 attempt 9.
• Every number on this card is a present-structure number. This run uses the zero-effect code (2026-10-15): dormant and dead units have scale 0, so they add nothing to the output and get no gradient. The checkpoint saves the scales, so it loads as the present structure. Use compact_model.build_from_plastic to build the physically smaller model.
• Present structure at step 10000: 8 layers (0-5, 8, 10), 36 of 144 heads, 357 of 1536 FFN groups. This is 0.425 of dense FLOPs/token and 0.432 of dense params.
• The gates of present units are 0.2-0.45, so architecture.json lists no unit at the 0.5 threshold. A present unit is a unit with a non-zero gate.
• The present structure writes fluent text (samples).

Full context: RESEARCH_LOG.md, the 2026-10-16 attempt-9 entries.

Intended use and limitations

• Intended use: reproduce and inspect the results of the Self-Organising Transformer study.
• Not for: production or downstream use. This is a small research model trained for 10,000 steps on WikiText-103 only. It is not instruction-tuned or safety-tuned.
• It needs the project code and transformers<5.0 (see How to use).

How to use

The model class is defined in the project repository and is not registered with AutoModel, so the Hub widget and AutoModelForCausalLM do not work. Use the repository code (it needs transformers<5.0; 5.x corrupts the reload of this model):

git clone https://github.com/sbartlett97/self-organising-transformer
cd self-organising-transformer && pip install -r requirements.txt
python generate.py sbartlett97/sot-phase2-wikitext103-heuristic-seed42-tc768d013-v9-zeroeffect --prompt "The history of the city begins in"

from ablation_importance import resolve_checkpoint
from evaluate import load_model
from transformers import AutoTokenizer

path = resolve_checkpoint("sbartlett97/sot-phase2-wikitext103-heuristic-seed42-tc768d013-v9-zeroeffect")
model = load_model(str(path), None).eval()
tok = AutoTokenizer.from_pretrained(path)
logits = model(tok("The history of the city begins in", return_tensors="pt").input_ids)["logits"]

Performance

No evaluate.py run. Final training-time val ppl of the present structure: 30.61 at 0.425 of dense FLOPs/token. Phase 1 dense 768 on the same method: 24.30 (1.0 of FLOPs). Dense controls at matched compute (0.42 of the FLOPs) do better: dense d 768 with 4 layers gets 30.18, dense d 512 with 9 layers gets 28.98. So Phase 2 has not yet shown that learned structure beats a fixed shape at the same compute (README).

Final present structure

8 of 12 layers present: 0.425 of dense FLOPs/token, 0.432 of dense parameters (compute.py). The checkpoint itself still holds every unit.

Structural plasticity events

• Total births: 92
• Total deaths: 60
• Total structural events: 152

Training trajectory

• Final train loss: 3.3856 at step 10000
• Final cost proxy (gate-weighted, not compute): 0.0345
• Final training-time val loss (random val batches): 3.4212 (ppl 30.61) at step 10000

Hyperparameters