Historic Language Models (HLMs)

Languages

Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:

Models

At the moment, the following models are available on the model hub:

We also released smaller models for the multilingual model:

Notice: We have released language models for Historic German and French trained on more noisier data earlier - see this repo for more information:

Corpora Stats

German Europeana Corpus

We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size and use less-noisier data:

For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:

French Europeana Corpus

Like German, we use different ocr confidence thresholds:

For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:

British Library Corpus

Metadata is taken from here. Stats incl. year filtering:

We use the year filtered variant. The following plot shows a tokens per year distribution:

Finnish Europeana Corpus

The following plot shows a tokens per year distribution:

Swedish Europeana Corpus

The following plot shows a tokens per year distribution:

All Corpora

The following plot shows a tokens per year distribution of the complete training corpus:

Multilingual Vocab generation

For the first attempt, we use the first 10GB of each pretraining corpus. We upsample both Finnish and Swedish to ~10GB. The following tables shows the exact size that is used for generating a 32k and 64k subword vocabs:

We then calculate the subword fertility rate and portion of [UNK]s over the following NER corpora:

Breakdown of subword fertility rate and unknown portion per language for the 32k vocab:

Breakdown of subword fertility rate and unknown portion per language for the 64k vocab:

Final pretraining corpora

We upsample Swedish and Finnish to ~27GB. The final stats for all pretraining corpora can be seen here:

Total size is 130GB.

Smaller multilingual models

Inspired by the "Well-Read Students Learn Better: On the Importance of Pre-training Compact Models" paper, we train smaller models (different layers and hidden sizes), and report number of parameters and pre-training costs:

We then perform downstream evaluations on the multilingual NewsEye dataset:

Pretraining

Multilingual model - hmBERT Base

We train a multilingual BERT model using the 32k vocab with the official BERT implementation on a v3-32 TPU using the following parameters:

python3 run_pretraining.py --input_file gs://histolectra/historic-multilingual-tfrecords/*.tfrecord \
--output_dir gs://histolectra/bert-base-historic-multilingual-cased \
--bert_config_file ./config.json \
--max_seq_length=512 \
--max_predictions_per_seq=75 \
--do_train=True \
--train_batch_size=128 \
--num_train_steps=3000000 \
--learning_rate=1e-4 \
--save_checkpoints_steps=100000 \
--keep_checkpoint_max=20 \
--use_tpu=True \
--tpu_name=electra-2 \
--num_tpu_cores=32

The following plot shows the pretraining loss curve:

Smaller multilingual models

We use the same parameters as used for training the base model.

hmBERT Tiny

The following plot shows the pretraining loss curve for the tiny model:

hmBERT Mini

The following plot shows the pretraining loss curve for the mini model:

hmBERT Small

The following plot shows the pretraining loss curve for the small model:

hmBERT Medium

The following plot shows the pretraining loss curve for the medium model:

English model

The English BERT model - with texts from British Library corpus - was trained with the Hugging Face JAX/FLAX implementation for 10 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:

python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-historic-english-cased/ \
--tokenizer_name /mnt/datasets/bert-base-historic-english-cased/ \
--train_file /mnt/datasets/bl-corpus/bl_1800-1900_extracted.txt \
--validation_file /mnt/datasets/bl-corpus/english_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 10 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-historic-english-cased-512-noadafactor-10e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length

The following plot shows the pretraining loss curve:

Finnish model

The BERT model - with texts from Finnish part of Europeana - was trained with the Hugging Face JAX/FLAX implementation for 40 epochs (approx. 1M steps) on a v3-8 TPU, using the following command:

python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-finnish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Finnish_0.6.txt \
--validation_file /mnt/datasets/hlms/finnish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-finnish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length

The following plot shows the pretraining loss curve:

Swedish model

The BERT model - with texts from Swedish part of Europeana - was trained with the Hugging Face JAX/FLAX implementation for 40 epochs (approx. 660K steps) on a v3-8 TPU, using the following command:

python3 run_mlm_flax.py --model_type bert \
--config_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--tokenizer_name /mnt/datasets/bert-base-swedish-europeana-cased/ \
--train_file /mnt/datasets/hlms/extracted_content_Swedish_0.6.txt \
--validation_file /mnt/datasets/hlms/swedish_validation.txt \
--max_seq_length 512 \
--per_device_train_batch_size 16 \
--learning_rate 1e-4 \
--num_train_epochs 40 \
--preprocessing_num_workers 96 \
--output_dir /mnt/datasets/bert-base-swedish-europeana-cased-512-dupe1-noadafactor-40e \
--save_steps 2500 \
--eval_steps 2500 \
--warmup_steps 10000 \
--line_by_line \
--pad_to_max_length

The following plot shows the pretraining loss curve:

Acknowledgments

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC) program, previously known as TensorFlow Research Cloud (TFRC). Many thanks for providing access to the TRC ❤️

Thanks to the generous support from the Hugging Face team, it is possible to download both cased and uncased models from their S3 storage 🤗