joseeangel/bert-base-uncased-conll2003-ner
bert-base-uncased adapted to Named entity recognition (CoNLL-2003) with Full fine-tuning.
Produced for the assignment U2T01 - Adapting BERT for NLP tasks (Trends in Data Science). The delivered method was chosen by measurement, not by default: the table below is the full set of adaptation methods trained for this task, and this repository holds the winner.
Intended use
Tagging English newswire text with PER / ORG / LOC / MISC entity spans in BIO format.
Out of scope: any use where an error carries real cost without a human in the loop, and any language or domain other than the one above.
Training data
- Dataset:
lhoestq/conll2003 - License: Reuters corpus terms / research use (CoNLL-2003 shared task)
- Citation: Tjong Kim Sang & De Meulder (2003), CoNLL-2003 shared task: language-independent NER
- Splits used: {"train": 14041, "dev": 3250, "test": 3453}
Adaptation method
| Method | Full fine-tuning |
| Trainable parameters | 108,898,569 of 108,898,569 (100.0%) |
| Head learning rate | 0.001 |
| Encoder learning rate | 2e-05 |
| Epochs / batch size | 3 / 32 |
| Max sequence length | 192 |
| Scheduler | linear with 10% warmup |
| Seed | 42 |
| Hardware | Apple M5 (mps) |
| Wall-clock training time | 10.6 min |
A freshly initialised head and pretrained encoder weights are trained in two parameter groups with separate learning rates; a single shared rate either starves the head or destroys pretrained features.
Evaluation
Held-out test split, never seen during training or model selection.
| Metric | Value |
|---|---|
| Entity F1 | 90.41 |
| Token accuracy | 98.08 |
What every method scored on this task
| Method | Trainable params | Share of model | Entity F1 | Token accuracy | Train time (min) |
|---|---|---|---|---|---|
| Feature-based (MLP probe) | 398,345 | 0.364% | 84.17 | 97.25 | 4.2 |
| Feature-based (linear probe) | 6,921 | 0.006% | 80.90 | 96.58 | 4.3 |
| Feature-based (logistic regression) | 6,921 | 0.0% | 77.84 | 96.10 | 1.2 |
| Partial FT (top 4 layers) | 28,358,409 | 26.041% | 88.10 | 97.71 | 4.6 |
| Full fine-tuning | 108,898,569 | 100.0% | 90.41 | 98.08 | 10.6 |
Single run per configuration with a fixed seed. Re-running with a different seed moves these numbers by roughly +/- 1-3 points, so gaps smaller than that are noise rather than findings.
Labels
0O1B-PER2I-PER3B-ORG4I-ORG5B-LOC6I-LOC7B-MISC8I-MISC
How to use
from transformers import pipeline
pipe = pipeline("token-classification", model="joseeangel/bert-base-uncased-conll2003-ner")
Limitations and bias
CoNLL-2003 is Reuters newswire from 1996-1997. The MISC class is heterogeneous and the weakest; entity types outside PER/ORG/LOC/MISC are not modelled; lowercase or noisy user-generated text degrades sharply because casing carries much of the signal. English only.
The model inherits whatever social bias is present in BERT's pretraining corpus (BooksCorpus + English Wikipedia) and in the task dataset above.
References
- Devlin, Chang, Lee & Toutanova (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 - section 5.3 is the feature-based vs fine-tuning comparison this work mirrors.
- Tjong Kim Sang & De Meulder (2003), CoNLL-2003 shared task: language-independent NER
- HuggingFace Transformers, fine-tune a pretrained model.