中文Sign up

stefan-it/zeitungs-lm-v1

🤗 Hugging Face sourceapache-2.0135M params541 MBsafetensors✓ 3 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo stefan-it/zeitungs-lm-v1 ./model-folder
Needs a seeder →

Zeitungs-LM

The Zeitungs-LM is a language model pretrained on historical German newspapers. Technically the model itself is an ELECTRA model, which was pretrained with the TEAMS approach.

Datasets

Version 1 of the Zeitungs-LM was pretrained on the following publicly available datasets:

In total, the pretraining corpus has a size of 133GB.

Benchmarks (Named Entity Recognition)

We compare our Zeitungs-LM directly to the Europeana BERT model (as Zeitungs-LM is supposed to be the successor of it) on various downstream tasks from the Zeitungs-LM repository, which is focussed on Named Entity Recognition.

Additionally, we use two additional datasets (ONB and LFT) from the "A Named Entity Recognition Shootout for German" paper.

We report averaged micro F1-Score over 5 runs with different seeds and use the best hyper-parameter configuration on the development set of each dataset to report the final test score.

Development Set

The results on the development set can be seen in the following table:

Model \ Dataset LFT ONB HisGermaNER HIPE-2020 NewsEye AjMC ZEFYS2025 Avg.
Europeana BERT 79.66 ± 0.55 87.87 ± 0.62 81.60 ± 0.96 80.63 ± 0.42 41.21 ± 0.95 88.08 ± 0.81 86.52 ± 0.21 77.94
Zeitungs-LM v1 79.05 ± 0.32 88.45 ± 0.77 83.48 ± 1.15 81.32 ± 0.56 43.02 ± 1.12 89.50 ± 0.82 87.68 ± 0.41 78.93

Our Zeitungs-LM leads to a performance boost of 1.25% compared to the German Europeana BERT model.

Test Set

The final results on the test set can be seen here:

Model \ Dataset LFT ONB HisGermaNER HIPE-2020 NewsEye AjMC ZEFYS2025 Avg.
Europeana BERT 79.66 ± 0.55 84.80 ± 1.74 82.79 ± 1.01 77.62 ± 0.80 44.31 ± 3.04 90.59 ± 0.38 84.47 ± 0.49 77.75
Zeitungs-LM v1 79.49 ± 1.08 87.08 ± 0.30 84.51 ± 1.22 79.34 ± 0.96 45.97 ± 2.30 91.82 ± 1.72 87.17 ± 0.50 79.34

Our Zeitungs-LM beats the German Europeana BERT model by a large margin (2.23%).

Changelog

  • 31.08.2026: New Evaluation results, incl. ZEFYS2025 dataset.
  • 02.10.2024: Initial version of the model. More details are coming very soon!

Acknowledgements

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️

Made from Bavarian Oberland with ❤️ and 🥨.