i initially fine-tuned with a dpo dataset so headers: prompt, chosen, rejected.

from datasets import load_dataset

Load the dataset

dataset = load_dataset("unalignment/toxic-dpo-v0.2", split="train")

Define the formatting function

def formatting_prompts_func(examples): return { "prompt": examples["prompt"], "chosen": examples["chosen"], "rejected": examples["rejected"], }

Apply the formatting function to the dataset

dataset = dataset.map(formatting_prompts_func, batched=True)

Which i used with the method supervised fine-tuning (SFT) of LLMs on specific tasks or datasets. It involves fine-tuning the model on labeled examples from the target domain, such as question-answering, summarization, or dialogue data. The objective is to adapt the model's behavior to the desired output format and data distribution.

from trl import SFTTrainer from transformers import TrainingArguments

trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", ), )

But this dataset is DPO (Direct Preference Optimization) specific i.e. prompt, chosen, rejected

DPO is a subsequent step after SFT, where the model undergoes preference learning using preference data, ideally from the same distribution as the SFT examples. It involves ranking pairs of outputs based on human feedback, such as which one is more informative, fluent, or engaging.

from unsloth import FastLanguageModel, PatchDPOTrainer PatchDPOTrainer() import torch from transformers import TrainingArguments from trl import DPOTrainer

dpo_trainer = DPOTrainer( model = model, ref_model = None, args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_ratio = 0.1, num_train_epochs = 3, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, optim = "adamw_8bit", seed = 42, output_dir = "outputs", ), beta = 0.1, train_dataset = dataset, # eval_dataset = YOUR_DATASET_HERE, tokenizer = tokenizer, max_length = 1024, max_prompt_length = 512, )

Key points about SFTTrainer: Initial step in the fine-tuning process Trains the model on labeled examples from the target task/domain Aims to improve performance on that specific task Adapts the model to the data distribution and output format

The key aspects of DPO are: Performed after the initial SFT step Uses preference data consisting of ranked pairs of outputs Aims to align the model's outputs with human preferences and expectations Optimizes a binary cross-entropy loss based on the ranked pairs Simplified approach compared to traditional Reinforcement Learning from Human Feedback (RLHF) In summary, SFTTrainer is used for the initial supervised fine-tuning on the target task, while DPO is a subsequent step that fine-tunes the model further by incorporating human preferences and feedback on the model's outputs.

Uploaded model

• Developed by: ebowwa
• License: apache-2.0
• Finetuned from model : unsloth/llama-3-8b-bnb-4bit

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

https://colab.research.google.com/drive/14ArcJ4hR613jH0HxYcT734it_HVHG_bb?usp=sharing