bert-base-cased for NER (full fine-tuning)

bert-base-cased fully fine-tuned for named entity recognition on CoNLL-2003. This is the delivered NER model: on the test set it scored higher than a feature-based run trained on the same data and seed.

Training data

• Dataset: lhoestq/conll2003 (CoNLL-2003).
• Splits: train for learning, validation for checkpoint selection, test reported once.
• Labels: O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC.
• The dataset stores ner_tags as integers. Names follow the standard CoNLL-2003 order.

Training

• Body: bert-base-cased (110M).
• Method: full fine-tuning. The encoder and the token-classification head both train.
• Learning rates: encoder 2e-5, classification head 1e-3.
• Epochs: 3. Batch size: 16. Max length: 128. Seed: 42.
• Subwords: the word label is placed on the first subword. Continuation pieces, [CLS], [SEP], and padding use -100 and are ignored by the loss.
• Metric: entity-level precision, recall, and F1 from seqeval. Token accuracy was logged and was not used to choose the model.

Metrics

Test set, one evaluation after training:

Validation entity F1 by epoch: 0.926, 0.944, 0.946. The saved checkpoint is the last epoch, which was also the best validation F1.

The rejected feature-based run (frozen BERT, linear head only) reached a test F1 of 0.7973 on the same split.

Intended use

English named entity recognition on sentences similar to CoNLL-2003 newswire: person, organization, location, and miscellaneous entities.

from transformers import AutoModelForTokenClassification, AutoTokenizer

repo_id = "karimepachecog/ner-bert-base-cased-full"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForTokenClassification.from_pretrained(repo_id)

Limitations

• Trained on English CoNLL-2003 newswire. It is not a general multilingual or domain-adapted NER system.
• A single run. Gaps of about 1–3 F1 points can come from the random seed.
• The label sits on the first subword only. Continuation pieces should not be read as a second entity.
• Entity types are the four CoNLL types. Other categories are marked O.

References

• Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. 2003.
• Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805
• Dataset: https://huggingface.co/datasets/lhoestq/conll2003
• Base model: https://huggingface.co/google-bert/bert-base-cased (Apache 2.0)