validation loss
| model | arxiv | github | books | wiki | webtext |
|---|---|---|---|---|---|
| horizon-pythia-ft-1.4b | 2.05 | 1.23 | 1.90 | 2.12 | 2.61 |
| pythia-1.4b | 2.13 | 1.25 | 1.91 | 2.18 | 2.62 |
horizon-derived reweighting media (mix-13)
| subset | documents |
|---|---|
| arxiv | 650 |
| github | 229 |
| books | 645 |
| wiki | 1560 |
| webtext | 10440 |