Fine-tuning-Gpt-2
LoRA adapter for openai-community/gpt2, fine-tuned with TRL SFT on TinyStories.
The base GPT-2 weights are unchanged. This repo contains the adapter only.
Training
- 500 steps on
train[:8000], sequence length 256 - LoRA rank 16, alpha 32, dropout 0.05, targets
c_attn,c_proj,c_fc - Batch size 1, gradient accumulation 8, learning rate 2e-4
- Final train loss 2.145, eval loss 1.986
Trained locally on an Apple M3 Pro.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "openai-community/gpt2"
adapter_id = "omaralshareef/Fine-tuning-Gpt-2"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(model, adapter_id)
prompt = "Once upon a time there was a little girl named Lucy."
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tokenizer.decode(output[0], skip_special_tokens=True))