German GPT-2 model

In this repository we release (yet another) GPT-2 model, that was trained on various texts for German.

The model is meant to be an entry point for fine-tuning on other texts, and it is definitely not as good or "dangerous" as the English GPT-3 model. We do not plan extensive PR or staged releases for this model 😉

Note: The model was initially released under an anonymous alias (anonymous-german-nlp/german-gpt2) so we now "de-anonymize" it.

More details about GPT-2 can be found in the great Hugging Face documentation.

Changelog

28.10.2026: Fix for the context length and the tokenizer configuration, see issue #20. Many thanks to Jochen for the investigation and the detailed bug report.

• The model was trained with a block size of 512 tokens, but config.json declared 1,024 positions. Position embeddings 512 to 1,023 were never trained, so per-token loss rose sharply from position 512 on. n_positions is now set to 512 and the untrained (all-zero) embedding rows were removed. Outputs for inputs of up to 512 tokens are identical to the previous revision. Inputs longer than 512 tokens now raise an error instead of silently returning wrong values. When computing perplexity or surprisal, please use windows of at most 512 tokens.
• The tokenizer now has a proper configuration: model_max_length is 512, so truncation=True works, and the special tokens are the ones that actually exist in the vocabulary (<s>, </s>, <pad>, <unk>). Previously, Transformers fell back to the GPT-2 default <|endoftext|> with id 50265, which lies outside the embedding matrix, while config.json pointed to an ordinary subword token (id 50256). Token ids of regular text are unchanged. Note that the model was trained on concatenated text without any separator token, so it never generates </s> on its own.

16.08.2021: Public release of re-trained version of our German GPT-2 model with better results.

15.11.2020: Initial release. Please use the tag v1.0 for this older version.

Training corpora

We use pretty much the same corpora as used for training the DBMDZ BERT model, that can be found in this repository.

Thanks to the awesome Hugging Face team, it is possible to create byte-level BPE with their awesome Tokenizers library.

With the previously mentioned awesome Tokenizers library we created a 50K byte-level BPE vocab based on the training corpora.

After creating the vocab, we could train the GPT-2 for German on a v3-8 TPU over the complete training corpus for 20 epochs. All hyperparameters can be found in the official JAX/FLAX documentation here from Transformers.

Using the model

The model itself can be used in this way:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dbmdz/german-gpt2")

model = AutoModelForCausalLM.from_pretrained("dbmdz/german-gpt2")

However, text generation is a bit more interesting, so here's an example that shows how to use the great Transformers Pipelines for generating text:

from transformers import pipeline

pipe = pipeline('text-generation', model="dbmdz/german-gpt2",
                 tokenizer="dbmdz/german-gpt2")

text = pipe("Der Sinn des Lebens ist es", max_length=100)[0]["generated_text"]

print(text)

This could output this beautiful text:

Der Sinn des Lebens ist es, im Geist zu verweilen, aber nicht in der Welt zu sein, sondern ganz im Geist zu leben.
Die Menschen beginnen, sich nicht nach der Natur und nach der Welt zu richten, sondern nach der Seele,'

License

All models are licensed under MIT.

Huggingface model hub

All models are available on the Huggingface model hub.

Contact (Bugs, Feedback, Contribution and more)

For questions about our German GPT-2 model just open an issue here 🤗

Acknowledgments

Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC). Thanks for providing access to the TFRC ❤️

Thanks to the generous support from the Hugging Face team, it is possible to download both cased and uncased models from their S3 storage 🤗