GPT-2 fine-tuned for apologies (Excusify)

Part of Excusify. Given a short prompt, the model writes an apology for a situation in an emotional, professional or informal tone.

Prompt format

professional | being late to the meeting :

The model writes the completion after the colon and stops with an end-of-text token.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sohamb2005/gpt2-finetuned-apologies"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

inputs = tokenizer("professional | being late to the meeting :", return_tensors="pt")
out = model.generate(**inputs, do_sample=True, top_p=0.92, temperature=0.8, max_new_tokens=60,
                     pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training


Base model	`gpt2` (124M parameters)
Data	`data/apologies.csv`: Type (emotional/professional/informal), Situation, Apology
Split	256 train / 29 held-out examples
Objective	causal LM loss on the completion and its end-of-text token (prompt masked)
Settings	learning rate 5e-05, 5 epochs, batch 8, cosine schedule; chosen from a sweep (best situation match, then lowest loss)

Evaluation

	Value
Held-out loss (GPT-2 before → this model)	3.298 → 1.86
Held-out perplexity (GPT-2 before → this model)	27.1 → 6.4
Situation match, held-out prompts	95% (hand-written answers: 96%)
Situation match, situations not in the data	100%

Situation match is the share of sampled answers that mention at least one content word of the situation in the prompt.

Limitations

Made for entertainment and practice messages. The training set is small and hand-written, so the model can repeat phrasings from it, and situations far from the training data may get generic answers.

加载中…