Training error in the weight-seed models

A training error caused the weight-seed models to continue training from the original Pythia weights with incorrect Adam optimizer buffers, rather than starting from independently initialized weights. These models therefore do not vary weight initialization and should not be used as weight-initialization ablations.

Corrected checkpoints are available at EleutherAI/pythia-160m-weight-seed1-corrected. This issue concerns the weight-seed models, not the data-seed models.

PolyPythias

This model is part of the PolyPythias suite, an extension of the Pythia project providing 45 additional training runs across 5 model sizes with 9 different random seeds each. These models enable systematic study of training stability and reproducibility in language models.

Paper

PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs

Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. ICLR 2025.

Model Details

All models were trained on 300B tokens from The Pile.

Naming Convention

• pythia-{size}m - Original Pythia model (seed 1234)
• pythia-{size}m-seed{1-9} - PolyPythias variants with different random seeds
• pythia-160m-data-seed{1-3} - 160M models with only data ordering varied (weight init fixed)
• pythia-160m-weight-seed{1-3} - Intended weight-initialization variants; affected by the training error described above.

The weight-seed releases do not provide the intended initialization ablation because of the training error described above.

Quick Start

from transformers import GPTNeoXForCausalLM, AutoTokenizer

# Load the final checkpoint
model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/pythia-70m-seed3")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-70m-seed3")

# Generate text
inputs = tokenizer("The quick brown fox", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0]))

Available Checkpoints

Each model provides 154 intermediate checkpoints saved as Git branches:

To load a specific checkpoint:

model = GPTNeoXForCausalLM.from_pretrained(
    "EleutherAI/pythia-70m-seed3",
    revision="step50000",  # Any checkpoint step
)

Training Data

All models were trained on The Pile using pre-shuffled data orderings. The shuffled index files for each seed are available at:

EleutherAI/pile-preshuffled-seeds

This dataset contains .idx files for seeds 0-9 used with MMapIndexedDataset to load the memory-mapped Pile data in the correct order for each seed.

Reproducing Training Data Order

To reproduce the exact data ordering used for a specific seed:

1. Download the Pile dataset and tokenize it using the Pythia tokenizer
2. Download the corresponding seed folder from pile-preshuffled-seeds:

   # Using huggingface_hub
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id="EleutherAI/pile-preshuffled-seeds",
       repo_type="dataset",
       allow_patterns="seed3/*",  # Download only seed3
       local_dir="./pile-seeds"
   )
   

3. Use the idx files with GPT-NeoX's MMapIndexedDataset:

   from dataset import MMapIndexedDataset
   dataset = MMapIndexedDataset(path_prefix, skip_warmup=True)
   

For complete training reproduction instructions, see the Pythia GitHub repository.

All PolyPythias Models

The complete collection is available at: EleutherAI/polypythias

14M Parameter Models

• pythia-14m-seed1 through pythia-14m-seed9

31M Parameter Models

• pythia-31m-seed1 through pythia-31m-seed9

70M Parameter Models

• pythia-70m-seed1 through pythia-70m-seed9

160M Parameter Models

• pythia-160m-seed1 through pythia-160m-seed9
• pythia-160m-data-seed1 through pythia-160m-data-seed3
• pythia-160m-weight-seed1 through pythia-160m-weight-seed3

410M Parameter Models

• pythia-410m-seed1 through pythia-410m-seed9

Evaluation Results

Evaluation results for all models are available in the polypythias-evals dataset.

Limitations

These models are released for research purposes only. They are not intended for deployment in production systems.

• Not instruction-tuned: These are base language models that predict the next token; they will not follow instructions like ChatGPT
• May generate harmful content: The Pile contains diverse internet text that includes biased, offensive, and factually incorrect content
• English only: Models were trained primarily on English text
• No safety filtering: Outputs are not filtered for safety or accuracy

License

Apache 2.0

Contact

For questions about these models, please use:

• EleutherAI Discord - #release-discussion channel
• GitHub Issues

Citation

If you use these models, please cite:

@inproceedings{vanderwal2025polypythias,
    title={PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs},
    author={van der Wal, Oskar and Lesci, Pietro and Muller-Eberstein, Max and Saphra, Naomi and Schoelkopf, Hailey and Zuidema, Willem and Biderman, Stella},
    booktitle={International Conference on Learning Representations},
    year={2025},
    url={https://arxiv.org/abs/2503.09543}
}