bert-base-agnews-topic-classification
Full fine-tuning of bert-base-uncased for 4-class news topic classification, delivered
as part of assignment U2T01 (Adapting BERT for NLP tasks — Trends in Data Science,
Unit 2, Universidad Politécnica de Yucatán).
Model description
bert-base-uncased with a linear classification head on top of the [CLS] token's
last hidden state, mapping to 4 topic classes: World, Sports, Business, Sci/Tech.
Training data
- Dataset: AG News (
fancyzhx/ag_news) - 10% of the official train split was held out as validation; the official test split was left untouched.
- 4 balanced classes.
Training procedure
Two adaptation methods were trained and compared; full fine-tuning is the delivered model, since it beat the feature-based baseline by a margin well above the run-to-run noise floor (±1–3 points from seed variance).
| Hyperparameter | Value |
|---|---|
| Base model | bert-base-uncased (110M params) |
| Method | Full fine-tuning (BERT body + head, jointly) |
| Learning rate (head) | 1e-3 |
| Learning rate (BERT body) | 2e-5 |
| Epochs | 2 |
| Batch size | 32 (train) / 64 (eval) |
| Seed | 42 |
| Trainable parameters | 109,485,316 |
| Training time | 76.4 min (single T4 GPU) |
Compared alternative (not delivered): feature-based adaptation — BERT body fully
frozen, with a scikit-learn logistic regression trained on the frozen [CLS]
embeddings (0 trainable BERT parameters, 17.7 min).
Evaluation results
| Method | Val Accuracy | Val F1 | Test Accuracy | Test F1 |
|---|---|---|---|---|
| Feature-based (LogReg on frozen [CLS]) | 90.60% | 90.56% | 90.24% | 90.23% |
| Full fine-tuning (delivered) | 94.92% | 94.89% | 94.59% | 94.60% |
The 4.36-point F1 gap on test is above the ±1–3 point noise margin observed for this setup, so it is treated as a real effect of the adaptation method rather than seed variance.
Intended uses & limitations
- Intended use: topic classification of short English news text into the 4 AG News categories, for coursework/research and as a fine-tuning reference example.
- Limitations: trained and evaluated only on AG News; it will not generalize to
topics or writing styles outside that domain (e.g. non-English text, other news
taxonomies, social media text). Trained for 2 epochs on a single seed — not
hyperparameter-tuned for production use. Inherits any biases present in the AG
News corpus and in
bert-base-uncased's pretraining data.
References
- Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
- Hugging Face. Fine-tune a pretrained model. https://huggingface.co/docs/transformers/training
- Dataset: fancyzhx/ag_news