TinyTransformer 5M - TinyStories

A 5M parameter TinyTransformer model trained on the TinyStories dataset.

This model is part of a research series investigating the effect of normalization strategies in small Transformer models. All models in the series use the same dataset and are trained under the same experimental setup, with normalization being the primary architectural variable.

Research Focus

The goal of this series is to compare different normalization strategies in small-scale Transformer architectures and evaluate their impact on training and model performance.

The series includes:

• Baseline - reference architecture
• Pre-LayerNorm - LayerNorm applied before the Transformer sublayers
• Post-LayerNorm - LayerNorm applied after the Transformer sublayers
• Pre-RMSNorm - RMSNorm applied before the Transformer sublayers
• Post-RMSNorm - RMSNorm applied after the Transformer sublayers

Models in This Series

Dataset

All models in this series were trained on:

TinyStories by Ronen Eldan and Yuanzhi Li

• Dataset: roneneldan/TinyStories
• Language: English
• Task: Masked language modeling

Normalization Methods

The experiments compare two commonly used normalization techniques:

Layer Normalization

Layer Normalization normalizes activations across the feature dimension and was introduced by Ba et al.

RMS Normalization

RMSNorm simplifies LayerNorm by removing the mean-centering operation and normalizing using the root mean square of the activations.

The experiments evaluate both methods in pre-normalization and post-normalization configurations.

Baseline

Just without any normalization

Citations

I'm used following papers in my reaserch:

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/abs/1706.03762

Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer Normalization. https://arxiv.org/abs/1607.06450

Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. https://arxiv.org/abs/1910.07467

License

All models in this series are released under the OpenMDW-1.1 license.

For the full license text, see the OpenMDW-1.1 license.