This is a decensored version of zerofata/MS3.2-PaintedFantasy-v2-24B, made using Heretic v1.2.0
Abliteration parameters
| Parameter | Value |
|---|---|
| direction_index | per layer |
| attn.o_proj.max_weight | 3.75 |
| attn.o_proj.max_weight_position | 26.51 |
| attn.o_proj.min_weight | 3.06 |
| attn.o_proj.min_weight_distance | 8.14 |
| mlp.down_proj.max_weight | 2.66 |
| mlp.down_proj.max_weight_position | 14.59 |
| mlp.down_proj.min_weight | 3.73 |
| mlp.down_proj.min_weight_distance | 25.63 |
Performance
| Metric | This model | Original model (zerofata/MS3.2-PaintedFantasy-v2-24B) |
|---|---|---|
| KL divergence | 1.4315 | 0 (by definition) |
| Refusals | 0/100 | 94/100 |
Painted Fantasy
PAINTED FANTASY v2
MS3.2-24BOverview
This is an uncensored creative model intended to excel at character driven RP / ERP.
Version 2 feels quite different from the original, with a heavy focus on reducing repetition across conversations and improving instruction following.
Has a pretty unique writing style and sense of creativity (IMO). Pays the price with intermittent brain farts though.
SillyTavern Settings
Recommended Roleplay Format
> Actions: In plaintext > Dialogue: "In quotes" > Thoughts: *In asterisks*Suggested Samplers
> Temp: 0.5-0.6 > MinP: 0.1 > TopP: 0.95 > Dry: 0.8, 1.75, 4Instruct
Mistral v7 Tekken
Quantizations
GGUF
> Static (mradermacher) > iMatrix (mradermacher)EXL3
> 3bpw > 3.5bpw > 4bpw > 5bpw > 6bpwTraining Process
Training process: SFT > DPO > KTO
SFT with RP/ERP, Stories and in character assistant data.
DPO focused on reducing repetition, misgendered characters and slop.
KTO focused on further reducing repetition and slop.
> Axolotl configsNot optimized for cost / performance efficiency, YMMV.
SFT 1*H100# ====================
# MODEL CONFIGURATION
# ====================
base_model: ConicCat/Mistral-Small-3.2-AntiRep-24B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template: mistral_v7_tekken
# ====================
# DATASET CONFIGURATION
# ====================
datasets:
- path: ./dataset.jsonl
type: chat_template
split: train
chat_template_strategy: tokenizer
field_messages: messages
message_property_mappings:
role: role
content: content
roles:
user: ["user"]
assistant: ["assistant"]
system: ["system"]dataset_prepared_path:
train_on_inputs: false # Only train on assistant responses
# ====================
# QLORA CONFIGURATION
# ====================
adapter: qlora
load_in_4bit: true
lora_r: 128
lora_alpha: 128
lora_dropout: 0.1
lora_target_linear: true
# lora_modules_to_save: # Uncomment only if you added NEW tokens
# ====================
# TRAINING PARAMETERS
# ====================
num_epochs: 3
micro_batch_size: 8
gradient_accumulation_steps: 1
learning_rate: 1e-5
optimizer: paged_adamw_8bit
lr_scheduler: rex
warmup_ratio: 0.05
weight_decay: 0.01
max_grad_norm: 1.0
# ====================
# SEQUENCE & PACKING
# ====================
sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
# ====================
# HARDWARE OPTIMIZATIONS
# ====================
bf16: auto
flash_attention: true
gradient_checkpointing: true
# ====================
# EVALUATION & CHECKPOINTING
# ====================
save_strategy: steps
save_steps: 20
save_total_limit: 5 # Keep best + last few checkpoints
load_best_model_at_end: true
metric_for_best_model: eval_loss
greater_is_better: false
# ====================
# LOGGING & OUTPUT
# ====================
output_dir: ./PT-SFT_1
logging_steps: 2
save_safetensors: true
# ====================
# WANDB TRACKING
# ====================
wandb_project: PF-SFT
wandb_entity: your_entity
wandb_name: run_name