Long-Horizon Historical BERT

Long-Horizon Historical BERT is a French historical language model based on dbmdz/bert-base-french-europeana-cased, adapted with temporal adapters trained on French public-domain historical newspapers.

The model is designed for experiments on historical NLP, especially where language use changes over time.

Model description

This model uses:

• base encoder: dbmdz/bert-base-french-europeana-cased
• task: masked language modeling
• adaptation method: temporal adapter bank
• training data: PleIAs/French-PD-Newspapers
• temporal periods:

PERIODS = [
    "pre_1850",
    "1850_1899",
    "1900_1938",
    "1939_1945",
    "post_1945",
]

Each input document is assigned to a temporal period based on its publication year. The corresponding adapter is applied after the BERT encoder.

How to load the model

This repository contains a custom PyTorch model. The file modeling_temporal_bert.py must be available locally.

import torch
from transformers import AutoTokenizer
from modeling_temporal_bert import HistoricalTemporalBertForMLM, year_to_period_id

model_id = "EmanuelaBoros/long-horizon-historical-bert"
base_model = "dbmdz/bert-base-french-europeana-cased"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = HistoricalTemporalBertForMLM(
    base_model_name=base_model,
    adapter_bottleneck_size=64,
    adapter_dropout=0.1,
)

state = torch.load("pytorch_model.bin", map_location="cpu")
model.load_state_dict(state, strict=False)
model.eval()

text = "Munich est mentionné dans le journal."
year = 1888

inputs = tokenizer(text, return_tensors="pt")
inputs["period_ids"] = torch.tensor([year_to_period_id(year)])

with torch.no_grad():
    outputs = model(**inputs)

Downstream evaluation: Named Entity Recognition

The model was evaluated on French HIPE-style historical newspaper NER data.

Dataset format

The NER data follows the HIPE TSV format:

TOKEN   NE-COARSE-LIT   NE-COARSE-METO   NE-FINE-LIT   NE-FINE-METO   NE-FINE-COMP   NE-NESTED   NEL-LIT   NEL-METO   MISC

For this evaluation, the label column was:

NE-COARSE-LIT

The data was split into sentences using MISC = EndOfSentence.

Data statistics

Labels:

{
  "O": 0,
  "B-loc": 1,
  "B-org": 2,
  "B-pers": 3,
  "B-prod": 4,
  "B-time": 5,
  "I-loc": 6,
  "I-org": 7,
  "I-pers": 8,
  "I-prod": 9,
  "I-time": 10
}

NER results

Three models were compared:

1. dbmdz/bert-base-french-europeana-cased
2. dbmdz/bert-base-french-europeana-cased with the year concatenated as a special prefix token
3. EmanuelaBoros/long-horizon-historical-bert

All models were fine-tuned for token classification on the same HIPE French NER data.

Overall test results

Development results

Test results by temporal period

Per-period F1 comparison

Per-class test results: baseline BERT

Per-class test results: year-prefix BERT

Per-class test results: long-horizon temporal BERT

Summary

On this HIPE French NER evaluation, the base Europeana BERT model obtains the best overall test F1:

0.7970 baseline vs 0.7884 year-prefix vs 0.7905 temporal

The year-prefix baseline does not improve overall performance, despite explicitly giving the publication year as input. The temporal model performs slightly better than the year-prefix baseline, but remains slightly below the base BERT model in this single run.

The per-period results show that temporal information has mixed effects. The year-prefix model performs best on 1900_1938, while the base BERT model is strongest on pre_1850, 1850_1899, and post_1945. The temporal adapter model is competitive across periods but does not clearly outperform the baseline in this run.

These results suggest that temporal adaptation preserves downstream NER performance, but more evaluation is needed to establish whether it improves robustness across periods. Future work should include multiple random seeds, additional HIPE label settings, and more historical NER datasets.

Citation

If you use this model, please cite the model repository and the underlying base model:

@misc{boros_long_horizon_historical_bert,
  title = {Long-Horizon Historical BERT},
  author = {Boros, Emanuela},
  year = {2026},
  howpublished = {https://huggingface.co/EmanuelaBoros/long-horizon-historical-bert}
}