I made another creating writing model.
But! Unlike the last couple! This one actually doesn't suck at RP or adventure! Hooray!
Usage notes: Chat template is ChatML, since this was trained using Muse-12B as the base model. Works with Temperature 1 / min-p 0.05, up through about Temperature 1.3 / min-p 0.02 if you like it hotter, probably works with other settings too.
Training notes: I started with Muse-12B because I like, in general, how this model behaves for creative uses. It's been my daily driver for a while, but I felt like the prose could stand to be a little more variable.
The training process went like this:
Stage 1:
- Data: A bunch of books (chosen for their prose/writing style). About 28M tokens per epoch.
- r32/a32 QLoRA at 32k context, applied only to the QKV tensors. LR: 1e-5, 2 epochs.
Stage 2:
- Trained on top of Stage 1.
- Data: RP data, about 4M tokens.
- r32/a32 QLoRA at 16k context, applied to
o_projanddown_projonly. LR: 5e-6, 1 epoch.
Stage 3:
- Trained on top of Stage 2.
- Data: Instruct data (1000 random samples from koto-instruct-sft), about 1.2M tokens.
- r32/a32 QLoRA at 4k context, applied to all linear modules. LR: 2e-6, 1 epoch.
Why?
Dunno. Fun experiment. Seemed to work, at least.
Enjoy!
(Big thanks to Gryphe and LatitudeGames for making Muse-12B. Great model, knocked it out of the park fr)