MN-12B-Nymphaea-RP

A fine-tune of Mistral Nemo Instruct 2407 for roleplay and creative writing.

I've trained this mostly for merging with Mistral Nemo fine-tunes.

[!Tip] The SillyTavern preset is available here. For custom presets, please use the Mistral V3-Tekken instruct template.

Training Notes

Trained on the latest iteration of my Darkmere dataset. This version features expanded genre variety, built upon a mix of manually curated synthetics and human-written stories.

[!IMPORTANT] The base weights are abliterated via Heretic prior to fine-tuning, so this fine-tune is quite uncensored.

Training Specs

Method:

• Training Method: DoRA (Weight-Decomposed LoRA)
• Target Modules all-linear
• LoRA Rank: 64
• LoRA Alpha: 64
• LoRA Dropout: 0.05

Hyperparameters:

• Batch Size: 2 (Per-device)
• Gradient Accumulation: 2
• Epochs: 2
• Learning Rate: 1e-4
• Optimizer: adamw_torch_fused
• LR Scheduler: cosine
• Noise Level: neftune_noise_alpha=5

Special Thanks

This fine-tune wouldn't be possible without the incredible work of the community:

• p-e-w for developing Heretic - an essential tool for censorship removal.
• SicariusSicariiStuff for developing SLOP_Detector script.
• Mistral AI for their Mistral Nemo weights.
• AMD for their Instinct™ MI300X GPU.