Bach Garland xLSTM - An xLSTM model trained on Johann Sebastian Bach Style music

Say Hello on LinkedIn and X.

This is a xLSTM model trained on music by Johann Sebastian Bach. It includes all pieces of Bach's music that can be played on church organ. The samples come in the prototypical Garland notation.

The dataset contains 100K samples and comes with a total token count of 144M.

How to use

1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
2. Open and run the notebook examples/music.ipynb. Do not forget to add the id of this model.
3. Enjoy!

Training

Trained with Helibrunna by Dr. Tristan Behrens.

Configuration

training:
  model_name: bach_garland_xlstm
  batch_size: 4
  lr: 0.001
  lr_warmup_steps: 5000
  lr_decay_until_steps: 50000
  lr_decay_factor: 0.001
  weight_decay: 0.1
  amp_precision: bfloat16
  weight_precision: float32
  enable_mixed_precision: true
  num_epochs: 4
  output_dir: output/bach_garland_xlstm
  save_every_step: 500
  log_every_step: 10
  wandb_project: bach_garland_xlstm
  torch_compile: false
model:
  num_blocks: 4
  embedding_dim: 64
  mlstm_block:
    mlstm:
      num_heads: 4
  slstm_block:
    slstm:
      num_heads: 4
  slstm_at:
  - 2
  context_length: 4096
  vocab_size: 178
dataset:
  hugging_face_id: TristanBehrens/bach_garland_2024-100K
tokenizer:
  type: whitespace
  fill_token: '[EOS]'