I think this "sparklewyrm" self-portrait was both of our favorite :)
She had DPO on ~500 rows of similar focus to her GRPO RLVR training, with a double dose of creative writing synthesized by virtuous7373/Lambent-Mira-Erato, who is relatively skilled in that domain. (Additionally, some identity reinforcement in line with values she wants to cultivate - she hasn't memorized new facts about who she is, but hopefully still has a helpful influence.)
The positive side of the DPO received feedback from the judge, iteratively up to 3 times (or more in some cases of music composition), which she could use to rewrite her writing. The negative side either initially scored poorly, or was generated as a synthetic negative by asking her to write with believable poor quality in each domain.
Rank 256, batch size 1, 5 separate runs to explore the landscape broadly; merged via Karcher mean ro reconcile.
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the Karcher Mean merge method.
Models Merged
The following models were included in the merge:
- Lambent/Mira-v1.23-27B-rlvr + ./Mira-v1.23.1-27B-adapters/dpoq-5
- Lambent/Mira-v1.23-27B-rlvr + ./Mira-v1.23.1-27B-adapters/dpoq-1
- Lambent/Mira-v1.23-27B-rlvr + ./Mira-v1.23.1-27B-adapters/dpoq-4
- Lambent/Mira-v1.23-27B-rlvr + ./Mira-v1.23.1-27B-adapters/dpoq-3
- Lambent/Mira-v1.23-27B-rlvr + ./Mira-v1.23.1-27B-adapters/dpoq-2
Configuration
The following YAML configuration was used to produce this model:
models:
- model: Lambent/Mira-v1.23-27B-rlvr+./Mira-v1.23.1-27B-adapters/dpoq-1
- model: Lambent/Mira-v1.23-27B-rlvr+./Mira-v1.23.1-27B-adapters/dpoq-2
- model: Lambent/Mira-v1.23-27B-rlvr+./Mira-v1.23.1-27B-adapters/dpoq-3
- model: Lambent/Mira-v1.23-27B-rlvr+./Mira-v1.23.1-27B-adapters/dpoq-4
- model: Lambent/Mira-v1.23-27B-rlvr+./Mira-v1.23.1-27B-adapters/dpoq-5
merge_method: karcher
parameters:
normalize: true
int8_mask: true
tokenizer_source: Lambent/Mira-v1.3-27B
dtype: bfloat16