Monolingual GPT-2 for Text Generation

This model is gpt2 (a decoder-only Transformer trained on English text), fine-tuned on the WikiText-2 dataset of Wikipedia articles. It takes the beginning of a sentence and continues it by predicting the next token, one at a time, using only the previous tokens. Because it is monolingual, it works with English only.

Details

Item	Value
Architecture	Decoder-only Transformer (GPT-2)
Base model	gpt2
Language	Monolingual (English)
Dataset	WikiText-2 (Wikipedia articles)
Task	Text generation
Training objective	Next-token prediction

Usage

from transformers import pipeline

gen = pipeline("text-generation", model="TahiraAbbas/gpt2-wikitext")
print(gen("The history of", max_new_tokens=40, do_sample=True)[0]["generated_text"])

Limitations

Fine-tuned on a small subset of WikiText-2. The text it generates can be repetitive or factually wrong, and it does not work for other languages.

Loading…