Bach Garland Pharia Plus - A Pharia model trained on Johann Sebastian Bach Style music

Say Hello on LinkedIn and X.

This is a Mamba model trained on music by Johann Sebastian Bach. It includes all pieces of Bach's music that can be played on church organ. The samples come in the prototypical Garland notation.

The dataset contains 100K samples and comes with a total token count of 144M.

How to use

1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
2. Open and run the notebook examples/music.ipynb. Do not forget to add the id of this model.
3. Enjoy!

Training

Trained with Helibrunna by Dr. Tristan Behrens.

Configuration

training:
  model_name: bach_garland_phariaplus
  batch_size: 22
  lr: 0.001
  lr_warmup_steps: 1818
  lr_decay_until_steps: 18181
  lr_decay_factor: 0.001
  weight_decay: 0.1
  amp_precision: bfloat16
  weight_precision: float32
  enable_mixed_precision: true
  num_epochs: 8
  output_dir: output/bach_garland_phariaplus
  save_every_step: 500
  log_every_step: 10
  wandb_project: bach_garland
  torch_compile: false
model:
  type: pharia
  attention_bias: true
  attention_dropout: 0.0
  eos_token_id: 0
  bos_token_id: 127179
  pad_token_id: 1
  hidden_act: gelu
  hidden_size: 132
  initializer_range: 0.02
  intermediate_size: 264
  max_position_embeddings: 2048
  mlp_bias: true
  num_attention_heads: 6
  num_hidden_layers: 6
  num_key_value_heads: 6
  rope_scaling: null
  rope_theta: 1000000
  tie_word_embeddings: false
  use_cache: true
  context_length: 2048
  vocab_size: 178
dataset:
  hugging_face_id: TristanBehrens/bach_garland_2024-100K
tokenizer:
  type: whitespace
  fill_token: '[EOS]'