| Tasks | Metric | Value | Stderr |
|---|---|---|---|
| MMLU (Average) | acc | 0.2282 | - |
| arc_challenge | acc_norm | 0.2210 | ± 0.0121 |
| arc_easy | acc_norm | 0.3279 | ± 0.0096 |
| boolq | acc | 0.3783 | ± 0.0085 |
| hellaswag | acc_norm | 0.2657 | ± 0.0044 |
| lambada_openai | acc / perplexity | 0.0470 / 13915.3 | ± 0.0029 / 752.0 |
| openbookqa | acc_norm | 0.2340 | ± 0.0190 |
| piqa | acc_norm | 0.5473 | ± 0.0116 |
| sciq | acc_norm | 0.4170 | ± 0.0156 |
| truthfulqa_mc1 | acc | 0.2815 | ± 0.0157 |
| truthfulqa_mc2 | acc | 0.5067 | ± 0.0160 |
| wikitext | byte / word ppl | 2.875 / 283.48 | N/A |
| winogrande | acc | 0.5012 | ± 0.0141 |