JS Fakes Music xLSTM - An xLSTM model trained on Johann Sebastian Bach Style music
This is an xLSTM trained on music. The dataset that has been used is JS Fakes Garland 100K, which is based on a collection of musical samples extracted from the JS Fake Chorales dataset by Omar Peracha. The samples come in the prototypical Garland notation.
The dataset contains 100K samples and comes with a total token count of 80M.
The model size is 138.78K trainable parameters.
How to use
- Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
- Open and run the notebook
examples/music.ipynb. - Enjoy!
Training
Trained with Helibrunna by Dr. Tristan Behrens.
Configuration
training:
model_name: jsfakes_garland_xlstm
batch_size: 16
lr: 0.001
lr_warmup_steps: 312
lr_decay_until_steps: 3125
lr_decay_factor: 0.001
weight_decay: 0.1
amp_precision: bfloat16
weight_precision: float32
enable_mixed_precision: true
num_epochs: 1
output_dir: output/jsfakes_garland_xlstm
save_every_step: 500
log_every_step: 10
wandb_project: jsfakes_garland_xlstm_2
torch_compile: false
model:
num_blocks: 4
embedding_dim: 64
mlstm_block:
mlstm:
num_heads: 4
slstm_block:
slstm:
num_heads: 4
slstm_at:
- 2
context_length: 2048
vocab_size: 115
modelGPT:
type: gpt2
num_blocks: 4
embedding_dim: 64
decoder:
num_heads: 4
context_length: 2048
dataset:
hugging_face_id: TristanBehrens/jsfakes_garland_2024-100K
tokenizer:
type: whitespace
fill_token: '[EOS]'