bert-base-cased for NER (feature-based)

bert-base-cased used as a frozen feature extractor for named entity recognition on CoNLL-2003. Only the token-classification head was trained. This run was kept as the rejected alternative: full fine-tuning scored higher on the same test set.

Training data

• Dataset: lhoestq/conll2003 (CoNLL-2003).
• Splits: train for learning, validation for checkpoint selection, test reported once.
• Labels: O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC.
• The dataset stores ner_tags as integers. Names follow the standard CoNLL-2003 order.

Training

• Body: bert-base-cased (110M), frozen. Trainable BERT parameters: 0. The head has 6,921 parameters.
• Method: feature-based adaptation. The encoder stayed in eval mode so dropout did not change the features. The linear head consumes the last hidden state.
• Learning rate: classification head 1e-3.
• Epochs: 3. Batch size: 16. Max length: 128. Seed: 42.
• Subwords: the word label is placed on the first subword. Continuation pieces, [CLS], [SEP], and padding use -100 and are ignored by the loss.
• Metric: entity-level precision, recall, and F1 from seqeval.

Metrics

Test set, one evaluation after training:

Validation entity F1 by epoch: 0.804, 0.820, 0.831. The saved checkpoint is the last epoch, which was also the best validation F1.

Full fine-tuning on the same data and seed reached a test F1 of 0.9114.

Intended use

English named entity recognition on sentences similar to CoNLL-2003 newswire, when a frozen encoder is required. For accuracy, prefer the full fine-tuning model.

from transformers import AutoModelForTokenClassification, AutoTokenizer

repo_id = "karimepachecog/ner-bert-base-cased-feature"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForTokenClassification.from_pretrained(repo_id)

Limitations

• Weaker than full fine-tuning on this task (test F1 0.7973 vs 0.9114). Validation F1 was still rising at epoch 3.
• The head classifies each token from the frozen last hidden state. It does not update BERT's contextual representations.
• Trained on English CoNLL-2003 newswire only.
• A single run. Gaps of about 1–3 F1 points can come from the random seed.
• The label sits on the first subword only.

References

• Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. 2003.
• Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805
• Dataset: https://huggingface.co/datasets/lhoestq/conll2003
• Base model: https://huggingface.co/google-bert/bert-base-cased (Apache 2.0)