AG News topic classification — full adaptation

Part of U2T01: Adapting BERT for NLP tasks. One of four models, each one a different head on the same bert-base-uncased body, each delivered with the adaptation method that its own measurements justified.

What this model does

Predicts World, Sports, Business, Sci/Tech.

How it was adapted

Full fine-tuning. Every encoder layer and the head were trained. The embedding matrix was kept frozen (23M of BERT's 110M parameters), which changes nothing measurable at these dataset sizes and saves optimiser memory.


Base model	`bert-base-uncased`
Adaptation	full
Trainable parameters	85,648,132 of 109,485,316 (78.23%)
Encoder layers trained	12 of 12
Head learning rate	0.001
Encoder learning rate	2e-05
Epochs	2
Batch size	32
Max sequence length	128
Seed	42
Training time	1.6 min on Tesla T4

A freshly initialised head and pretrained encoder layers were given separate learning rates through two optimiser parameter groups; a single learning rate for both silently cripples this kind of run.

Training data

AG News (fancyzhx/ag_news). Training examples used: 6000.

Evaluation

Metric	Test	Validation
loss	28.15	27.88
accuracy	91.45	90.85
macro_f1	91.22	90.81
macro_precision	91.31	90.91
macro_recall	91.19	90.83

Metric choice is argued in the project report. In short: entity-level F1 for NER because ~83% of CoNLL tokens are O and token accuracy would flatter a model that predicts nothing; token accuracy for POS because there is no dominant class; EM and F1 for SQuAD because each covers the other's blind spot.

Why this method, and what lost

Run	Method	Trainable params	accuracy (test)
`agnews_feature_logreg_fast`	feature	0	86.35
`agnews_feature_logreg_mini`	feature	0	84.20
`agnews_feature_mlp_fast`	feature	0	89.35
`agnews_feature_mlp_mini`	feature	0	87.10
`agnews_full_fast`	full	85,648,132	91.45
`agnews_full_mini`	full	85,648,132	83.40

Decision rule: best primary metric on test, and where two methods fall inside a ±1 point band — smaller than the ±1–3 points a different random seed moves results by — the cheaper one wins.

Intended use and limitations

Intended for research and coursework: benchmarking adaptation strategies and as a starting point for related English tasks.

Not intended for production decisions about people. Concretely:

English only, and the training text is narrow. AG News is 2004 newswire; CoNLL-2003 is Reuters newswire, so it under-performs on social media, names outside Western European conventions, and any entity that became prominent after 2003. UD EWT is web English (blogs, reviews, emails). SQuAD contexts are Wikipedia paragraphs, and the model always returns a span — SQuAD v1.1 contains no unanswerable questions, so it cannot say "I don't know".
Single run, single seed. Re-running with a different seed moves results by roughly ±1–3 points. Differences smaller than that are not meaningful.
Inherited bias. bert-base-uncased carries the associations of its pretraining corpus, and nothing here mitigates them.
Sub-sampled training data, so these numbers sit below published full-data results. Every adaptation method on this task saw exactly the same data, so the comparison that chose this method is unaffected.

Reproduce it

git clone <this project>
pip install -r requirements.txt
python -m u2t01.cli run configs_fast/agnews_full.yaml

Fixed seed (42), pinned dependencies, and one YAML file per experiment: config + seed + requirements.txt fully determine a run.

References

Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 — https://arxiv.org/abs/1810.04805 (§5.3 is the feature-based vs fine-tuning comparison this project mirrors)
Tunstall, von Werra & Wolf, Natural Language Processing with Transformers, O'Reilly, chapters 1–3
Hugging Face, Fine-tune a pretrained model — https://huggingface.co/docs/transformers/training
Dataset: fancyzhx/ag_news

Card generated 2026-09-27 from results/agnews_full_fast.json.

toyin88/u2t01-bert-agnews