BEE-spoke-data/NanoLlama-GQA-L10-A32_KV8-v13-KI
note that training still WIP
This model is a fine-tuned version of BEE-spoke-data/NanoLlama-GQA-L10-A32_KV8-v12-minipile on the None dataset. It achieves the following results on the evaluation set:
- Loss: 2.5937
- Accuracy: 0.4948
Training and evaluation data
KI dataset
hf-causal-experimental (pretrained=BEE-spoke-data/NanoLlama-GQA-L10-A32_KV8-v13-KI,revision=main,trust_remote_code=True,dtype='float'), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| arc_easy | 0 | acc | 0.4322 | ± | 0.0102 |
| acc_norm | 0.3960 | ± | 0.0100 | ||
| boolq | 1 | acc | 0.6196 | ± | 0.0085 |
| lambada_openai | 0 | ppl | 61.6595 | ± | 2.4362 |
| acc | 0.2779 | ± | 0.0062 | ||
| openbookqa | 0 | acc | 0.1540 | ± | 0.0162 |
| acc_norm | 0.2840 | ± | 0.0202 | ||
| piqa | 0 | acc | 0.6028 | ± | 0.0114 |
| acc_norm | 0.6028 | ± | 0.0114 | ||
| winogrande | 0 | acc | 0.5193 | ± | 0.0140 |
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.00025
- train_batch_size: 8
- eval_batch_size: 4
- seed: 2280
- gradient_accumulation_steps: 16
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 2.5744 | 0.08 | 200 | 2.7138 | 0.4776 |
| 2.5387 | 0.16 | 400 | 2.6713 | 0.4836 |
| 2.4718 | 0.23 | 600 | 2.6462 | 0.4873 |
| 2.4681 | 0.31 | 800 | 2.6328 | 0.4892 |
| 2.5351 | 0.39 | 1000 | 2.6227 | 0.4908 |
| 2.5316 | 0.47 | 1200 | 2.6159 | 0.4914 |
| 2.527 | 0.54 | 1400 | 2.6103 | 0.4921 |
| 2.4838 | 0.62 | 1600 | 2.6058 | 0.4930 |
| 2.4483 | 0.7 | 1800 | 2.6024 | 0.4934 |
| 2.426 | 0.78 | 2000 | 2.5998 | 0.4937 |
| 2.4685 | 0.86 | 2200 | 2.5961 | 0.4944 |
| 2.4473 | 0.93 | 2400 | 2.5937 | 0.4948 |
Framework versions
- Transformers 4.36.0.dev0
- Pytorch 2.1.0
- Datasets 2.15.0
- Tokenizers 0.15.0
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 29.23 |
| AI2 Reasoning Challenge (25-Shot) | 23.81 |
| HellaSwag (10-Shot) | 29.39 |
| MMLU (5-Shot) | 25.37 |
| TruthfulQA (0-shot) | 44.77 |
| Winogrande (5-shot) | 51.14 |
| GSM8k (5-shot) | 0.91 |