nanochat-jpを用いて22層のtransformerをフルスクラッチから事前学習及び教師あり微調整 (SFT) を実施した際のトークン分割器とチェックポイントです.
事前学習データにはnanochat-jp v4データセットを利用しました.
教師あり微調整 (SFT) にはnanochat-jp-sftを利用しました.
環境変数NANOCHAT_BASE_DIRに本リポジトリのダウンロード先を指定することで,追加学習等を実施することができます.
本リポジトリでのモデルの学習には玄界 (ノードB) 1ノードを24時間利用し学習を実施しました.
base_checkpoints/d22/model_005000.pt 3.4 GBbase_checkpoints/d22/model_010000.pt 3.4 GBbase_checkpoints/d22/model_015000.pt 3.4 GBbase_checkpoints/d22/model_020000.pt 3.4 GBbase_checkpoints/d22/model_025000.pt 3.4 GBbase_checkpoints/d22/model_030000.pt 3.4 GBbase_checkpoints/d22/model_032587.pt 3.4 GBchatsft_checkpoints/d22/model_002416.pt 3.4 GBbase_checkpoints/d22/optim_005000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_005000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_005000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_005000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_010000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_010000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_010000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_010000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_015000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_015000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_015000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_015000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_020000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_020000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_020000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_020000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_025000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_025000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_025000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_025000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_030000_rank0.pt 1.2 GBbase_checkpoints/d22/optim_030000_rank1.pt 1.2 GBbase_checkpoints/d22/optim_030000_rank2.pt 1.2 GBbase_checkpoints/d22/optim_030000_rank3.pt 1.2 GBbase_checkpoints/d22/optim_032587_rank0.pt 1.2 GBbase_checkpoints/d22/optim_032587_rank1.pt 1.2 GBbase_checkpoints/d22/optim_032587_rank2.pt 1.2 GBbase_checkpoints/d22/optim_032587_rank3.pt 1.2 GBchatsft_checkpoints/d22/optim_002416_rank0.pt 1.2 GBchatsft_checkpoints/d22/optim_002416_rank1.pt 1.2 GBchatsft_checkpoints/d22/optim_002416_rank2.pt 1.2 GBchatsft_checkpoints/d22/optim_002416_rank3.pt 1.2 GBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo tohoku-nlp/nanochat-jp-v4-d22 ./model-folder