music-generation

This model is a fine-tuned version of gpt2 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5312

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 256
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 20
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
3.7238 0.9217 100 2.8460
2.4643 1.8387 200 1.8829
1.8339 2.7558 300 1.4234
1.5013 3.6728 400 1.2203
1.3125 4.5899 500 1.0966
1.1899 5.5069 600 1.0028
1.0982 6.4240 700 0.9353
1.0302 7.3410 800 0.8779
0.9766 8.2581 900 0.8276
0.9243 9.1751 1000 0.7757
0.8825 10.0922 1100 0.7345
0.845 11.0092 1200 0.7000
0.8083 11.9309 1300 0.6624
0.7784 12.8479 1400 0.6328
0.7502 13.7650 1500 0.6052
0.7281 14.6820 1600 0.5816
0.7072 15.5991 1700 0.5622
0.6903 16.5161 1800 0.5486
0.6796 17.4332 1900 0.5386
0.6705 18.3502 2000 0.5335
0.6646 19.2673 2100 0.5312

Framework versions

  • Transformers 4.55.0
  • Pytorch 2.6.0+cu124
  • Datasets 4.0.0
  • Tokenizers 0.21.4