GPT-2

Pretrained model on Bulgarian language using a causal language modeling (CLM) objective. It was introduced in this paper and first released at this page.

Model description

This is the SMALL version compressed via progressive module replacing.

The compression was executed on Bulgarian text from OSCAR, Chitanka and Wikipedia.

Intended uses & limitations

You can use the raw model for:

• text generation
• auto-complete
• spelling correction

Or fine-tune it to a downstream task.

How to use

Here is how to use this model in PyTorch:

>>> from transformers import AutoModel, AutoTokenizer
>>>
>>> model_id = "rmihaylov/gpt2-small-theseus-bg"
>>> tokenizer = AutoTokenizer.from_pretrained(model_id)
>>> model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
>>>
>>> input_ids = tokenizer.encode(
>>>     "Здравей,", 
>>>     add_special_tokens=False, 
>>>     return_tensors='pt')
>>>
>>> output_ids = model.generate(
>>>     input_ids, 
>>>     do_sample=True, 
>>>     max_length=50, 
>>>     top_p=0.92, 
>>>     pad_token_id=2,
>>>     top_k=0)
>>>
>>> output = tokenizer.decode(output_ids[0])
>>>
>>> output = output.replace('<|endoftext|>', '\n\n\n')
>>> output = output.replace('<|unknown|>', '')
>>> output = output.replace('▁', ' ')
>>> output = output.replace('<|n|>', '\n')
>>>
>>> print(output)

Здравей, извинявай, но не мога да заспя. 
 Джини се обърна и забеляза колко са прегърнати. 
 — Почакай, Джини. Не мога да повярвам, че е възможно! Толкова искам да те видя. 
 — Обеща

Limitations and bias

As the openAI team themselves point out in their model card:

Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.

Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.