GPT-2 Roman Urdu Sentiment Analysis

Fine-tuned gpt2 (decoder-only, causal transformer, 124M parameters) with a sequence classification head for 3-class sentiment (negative / neutral / positive) on Roman Urdu text.

Architecture

GPT-2 is a decoder-only transformer with causal (left-to-right) attention. For classification, the hidden state of the last non-pad token is passed to a linear layer.

Dataset

community-datasets/roman_urdu (about 20k sentences), stratified 80/10/10 train/val/test split, same split as the BERT model.

Fine-tuning

Epochs: 3, learning rate: 5e-05, batch size: 16, max length: 128, optimizer: AdamW, mixed precision (fp16).

Results (test set)

• Accuracy: 0.623
• Macro-F1: 0.610
• BERT (encoder-only) for comparison: Accuracy 0.644, Macro-F1 0.632

Limitations

GPT-2 was pretrained mostly on English text, so Roman Urdu spelling variations and slang can be misclassified.