BERT adapted to named entity recognition (CoNLL-2003, IOB2) — full fine-tuning

abelcetina/u2t01-bert-conll2003-ner — bert-base-uncased adapted to named entity recognition (CoNLL-2003, IOB2) with full fine-tuning.

Produced for the university assignment U2T01: Adapting BERT for NLP tasks. The companion repository contains the training code, the raw results file behind every number below, and the comparison against the alternative adaptation methods.

Model description

• Base encoder: bert-base-uncased (108,898,569 parameters in total).
• Task: named entity recognition (CoNLL-2003, IOB2).
• Adaptation method: full fine-tuning. Every parameter of the encoder and of the task head received gradients, with a small learning rate on the body and a larger one on the randomly initialised head.
• Head: a linear token-classification head over every token representation.
• Label set: 9 IOB2 tags: O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC
• Trained by: U2T01 team · generated 2026-09-25.

Intended use

• Research and teaching: reproducing the feature-based versus fine-tuning comparison of BERT section 5.3 across four task types.
• Inference on English text that resembles the training domain (Reuters newswire text from 1996–1997).
• Usage:

from transformers import pipeline
tagger = pipeline("token-classification", model="abelcetina/u2t01-bert-conll2003-ner", aggregation_strategy="simple")
print(tagger("Angela Merkel visited the European Central Bank in Frankfurt."))

Out-of-scope use

• Any decision affecting a person (hiring, moderation with consequences, credit, legal or medical decisions). This is a coursework model validated on one academic benchmark with a single seed.
• Languages other than English, and domains far from Reuters newswire text from 1996–1997 — performance is not characterised there and is expected to drop.
• Extracting facts about the world: for the QA model, the answer is a span copied from the context passage it is given, never knowledge the model holds.
• Commercial redistribution without checking the training-data licence below.

Training data

Read from a parquet mirror rather than from the legacy script-based conll2003, because datasets >= 4.0 removed dataset loading scripts.

Adaptation method and what was frozen

• Method flag: --method full
• Trainable components: the whole BERT encoder (embeddings + 12 layers) and the task head
• Frozen components: nothing

Hyper-parameters

Evaluation results

Metrics computed by src/metrics.py; the raw values live in ner__full__seed42.json.

Evaluation split (validation)

Test split

Headline number: f1 = 0.9395 (accuracy = 0.9882).

The comparison against the other adaptation methods of this task, with the gap in percentage points and the noise flag, is in report/results_tables.md of the companion code repository (Table 1, Table 2.ner and Table 3). A gap below one point there is reported as noise, not as an improvement.

Limitations and bias

• Dataset bias. Reuters newswire from the late 1990s: person, organisation and location names are heavily skewed towards the actors and the spelling conventions of that period and towards sports reporting, which dominates the corpus. The model inherits both that domain skew and the social biases of the bert-base-uncased pre-training corpus (BookCorpus and English Wikipedia), which is known to encode gender, ethnic and occupational stereotypes. Nothing here mitigates or measures them.
• Single seed. Every number on this card comes from one run with seed 42. No variance estimate exists, so a difference of roughly one point against another configuration must be read as noise rather than as an improvement.
• Subsampling. Training used 14,041 examples (hyperparams.train_size). Read from a parquet mirror rather than from the legacy script-based conll2003, because datasets >= 4.0 removed dataset loading scripts.
• Sequence truncation. Inputs were truncated to 128 tokens; longer inputs lose their tail at inference too.
• Frozen-encoder caveat (feature-based models only). Not applicable: this model was fine-tuned, so the encoder weights moved.
• Only a validation (and where available test) split was measured. No robustness, adversarial, out-of-domain or fairness evaluation was performed.

Environmental and compute footprint

A single short fine-tuning run on one T4 is a small footprint in absolute terms, and that is the point of this assignment: the cost-versus-quality table in the companion report shows how much of the quality survives when only a fraction of the parameters is trained, which is the part that scales. No CO₂e figure is claimed here, because the energy mix of the Colab region is unknown.

References and citation

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT. arXiv:1810.04805. Section 5.3 is the feature-based versus fine-tuning comparison this model is part of.
• Tjong Kim Sang, E. F., & De Meulder, F. (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. CoNLL. https://aclanthology.org/W03-0419/
• Tunstall, L., von Werra, L., & Wolf, T. (2022). Natural Language Processing with Transformers, chapters 1–3. O'Reilly.
• Hugging Face, Fine-tuning a pretrained model: https://huggingface.co/docs/transformers/training

@inproceedings{devlin2019bert,
  title     = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {Proceedings of NAACL-HLT},
  pages     = {4171--4186},
  year      = {2019},
  url       = {https://arxiv.org/abs/1810.04805}
}

Cite this adapted model as:

@misc{u2t01_ner_full,
  title  = {BERT adapted to named entity recognition (CoNLL-2003, IOB2) — full fine-tuning},
  author = {U2T01 team},
  year   = {2026},
  note   = {U2T01: Adapting BERT for NLP tasks. Adaptation method: full. Seed 42.},
  url    = {https://huggingface.co/abelcetina/u2t01-bert-conll2003-ner}
}