CoNLL-2003 named entity recognition — feature adaptation

Part of U2T01: Adapting BERT for NLP tasks. One of four models, each one a different head on the same bert-base-uncased body, each delivered with the adaptation method that its own measurements justified.

This repository is not a fine-tuned transformer. BERT was used as a frozen feature extractor, so there is no task head inside the model weights. The repo contains the unmodified bert-base-uncased body, the fitted scikit-learn probe as probe.joblib, and inference.py showing the two-step call. AutoModelFor... will load the body but will not classify.

What this model does

Predicts O, B/I-PER, B/I-ORG, B/I-LOC, B/I-MISC (IOB2).

How it was adapted

Feature-based adaptation. The BERT body was frozen completely — zero of its parameters were updated. Its hidden states were extracted once and a classical classifier was fitted on top.

A freshly initialised head and pretrained encoder layers were given separate learning rates through two optimiser parameter groups; a single learning rate for both silently cripples this kind of run.

Training data

CoNLL-2003 (lhoestq/conll2003). Training examples used: 1500.

Evaluation

Metric choice is argued in the project report. In short: entity-level F1 for NER because ~83% of CoNLL tokens are O and token accuracy would flatter a model that predicts nothing; token accuracy for POS because there is no dominant class; EM and F1 for SQuAD because each covers the other's blind spot.

Why this method, and what lost

Decision rule: best primary metric on test, and where two methods fall inside a ±1 point band — smaller than the ±1–3 points a different random seed moves results by — the cheaper one wins.

Intended use and limitations

Intended for research and coursework: benchmarking adaptation strategies and as a starting point for related English tasks.

Not intended for production decisions about people. Concretely:

• English only, and the training text is narrow. AG News is 2004 newswire; CoNLL-2003 is Reuters newswire, so it under-performs on social media, names outside Western European conventions, and any entity that became prominent after 2003. UD EWT is web English (blogs, reviews, emails). SQuAD contexts are Wikipedia paragraphs, and the model always returns a span — SQuAD v1.1 contains no unanswerable questions, so it cannot say "I don't know".
• Single run, single seed. Re-running with a different seed moves results by roughly ±1–3 points. Differences smaller than that are not meaningful.
• Inherited bias. bert-base-uncased carries the associations of its pretraining corpus, and nothing here mitigates them.
• Sub-sampled training data, so these numbers sit below published full-data results. Every adaptation method on this task saw exactly the same data, so the comparison that chose this method is unaffected.

Reproduce it

git clone <this project>
pip install -r requirements.txt
python -m u2t01.cli run configs_mini/ner_feature.yaml

Fixed seed (42), pinned dependencies, and one YAML file per experiment: config + seed + requirements.txt fully determine a run.

References

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 — https://arxiv.org/abs/1810.04805 (§5.3 is the feature-based vs fine-tuning comparison this project mirrors)
• Tunstall, von Werra & Wolf, Natural Language Processing with Transformers, O'Reilly, chapters 1–3
• Hugging Face, Fine-tune a pretrained model — https://huggingface.co/docs/transformers/training
• Dataset: lhoestq/conll2003

Card generated 2026-09-27 from results/ner_feature_mini.json.