This is a part of stories-llama2-* model family:
| name | params | layers | hidden_size | query heads | key & value heads |
|---|---|---|---|---|---|
| stories-llama2-50k | 49,554 | 1 | 6 | 3 | 1 |
| stories-llama2-100k | 99,924 | 1 | 12 | 2 | 1 |
| stories-llama2-250k | 246,820 | 2 | 28 | 2 | 1 |
| stories-llama2-500k | 527,912 | 2 | 56 | 4 | 2 |
| stories-llama2-1m | 1,019,508 | 4 | 84 | 6 | 3 |
| stories-llama2-2.5m | 2,437,280 | 4 | 160 | 8 | 4 |
| stories-llama2-5m | 5,136,720 | 5 | 240 | 10 | 5 |
| stories-llama2-10m | 10,421,340 | 6 | 340 | 10 | 5 |
| stories-llama2-25m | 24,215,520 | 8 | 480 | 16 | 8 |
| stories-llama2-50m | 49,387,712 | 8 | 704 | 16 | 8 |
You can access W&B logs here.
This model was trained using delphi. See training_config.json and run_context.json for details.