Model Card for Model ID

ModernBERT model trained on Bulgarian literature, Web, and other datasets - uncased.

Model Details

124M parameter ModernBERT model trained on 29B (35B depending on tokenization) tokens for 3 epochs with Masked Language Modelling objective.

• Tokenizer vocabulary size is 50176.

• Model hidden dimension is 768.

• Feed-Forward dimension is 1152.

• Hidden layer count is 22.

• Developed by: Artificial Inteligence and Language Technologies Department at Institute of Information and Communication Technologies - Bulgarian Academy of Sciences.

• Funded by: The model is pretrained within the CLaDA-BG: National Interdisciplinary Research E-Infrastructure for Bulgarian Language and Cultural heritage - member of the pan-European research consortia CLARIN-ERIC & DARIAH-ERIC, funded by the Ministry of Education and Science of Bulgaria (support for the Bulgarian National Roadmap for Research Infrastructure). The training was performed at the supercomputer HEMUS at IICT-BAS, part of the RIs of the CoE on Informatics and ICT, financed by the OP SESG (2014–2020), and co-financed by the European Union through the ESIF.

• Model type: ModernBERT

• Language(s) (NLP): Bulgarian.

• License: MIT

Uses

The model is intended to be used as a base model for fine-tuning tasks in NLP.

Direct Use

>>> from transformers import (
>>>     PreTrainedTokenizerFast,
>>>     ModernBertForMaskedLM,
>>>     pipeline 
>>> )

>>> model = ModernBertForMaskedLM.from_pretrained('AIaLT-IICT/modern_bert_bg_base_uncased')
>>> tokenizer = PreTrainedTokenizerFast.from_pretrained('AIaLT-IICT/modern_bert_bg_base_uncased')

>>> fill_mask = pipeline(
>>>     "fill-mask",
>>>     model=model,
>>>     tokenizer=tokenizer
>>> )


>>> fill_mask("Заради 3 завода няма да [MASK] нито есенниците неподхранени, нито зърното да поскъпне заради тях.")

[{'score': 0.4933295249938965,
  'token': 17875,
  'token_str': 'останат',
  'sequence': 'заради 3 завода няма да останат нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.06887241452932358,
  'token': 12941,
  'token_str': 'има',
  'sequence': 'заради 3 завода няма да има нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.06286628544330597,
  'token': 13311,
  'token_str': 'бъдат',
  'sequence': 'заради 3 завода няма да бъдат нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.05207058787345886,
  'token': 26913,
  'token_str': 'оставим',
  'sequence': 'заради 3 завода няма да оставим нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.05052710324525833,
  'token': 12861,
  'token_str': 'са',
  'sequence': 'заради 3 завода няма да са нито есенниците неподхранени, нито зърното да поскъпне заради тях.'}]

Out-of-Scope Use

The model is not trained on Next Sentence prediction so the [CLS] token embedding will not be useful out of the box. If you want to use the model for Sequence classification it is recommended to fine-tune it.

Recommendations

It is recommended to use the model for Token Classification and Sequence classification fine-tuning tasks. The model can be used within SentenceTransformers framework for producing embeddings.

Training Details

Training Data

Trained on 29B tokens consisting of deduplicated union of:

• uonlp/CulturaX
• MaCoCu-bg 2.0
• HPLT 2.0 Bulgarian (Cyrillic) cleaned
• Literature
• Wikipedia
• others

Training Procedure

Trained with Masked Language Modelling with 20% masks for 3 epochs with bf16 mixed precision, 512 tokens context and batch size of 256*512 tokens. After this training session, the context length was extended to 8,192 tokens through additional training with a batch size of 128 tokens and approximately 900K documents with longer contexts, filtered from the training data, totaling about 7B words.

Evaluation

The model is evaluated on the Masked Language Modelling objective on test split with 20% random masked tokens. It achieves test loss of 0.99 and test accuracy of 77.58%

Citation

@inproceedings{simov-etal-2026-small,
  title = {Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian },
  author = {Simov, Kiril and Paev, Nikolay and Osenova, Petya and Valchev, Teodor and Marinov, Stefan},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  month = {May},
  year = {2026},
  pages = {187--197},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Montejo-Raez, Arturo and Grisot, Cristina and Blochowiak, Joanna and Ljubešić, Nikola and Battaner, Elena and Rigau, German},
  doi = {10.63317/5n9bq7g3it2s},
  abstract = {In the paper we present a set of small LLM-based models for solving the basic NLP tasks for Bulgarian - POS tagging, Lemmatization, Dependency parsing, Named Entity Recognition, Named Entity Linking, Event Annotation, among others. In order to create fine-tuned models for these tasks, we first pre-train models using architectures like BERT, Modern-BERT, and T5 with different sizes, over Bulgarian data only. For each of the tasks we report our approach towards the fine-tuning, the results from the experiments and also the evaluation. Then we define a way to visualize the results over HTML documents which contain the analyzed texts. Our rationale are as follows: most, if not all SSH research scenarios, need a reliable processing chains that can be customized with respect to the specific needs. These scenarios also would need proper visualization for human observation. We aim to provide such a basic LLM-based toolkit.}
}

Model Card Authors

Nikolay Paev, Kiril Simov

Model Card Contact

nikolay.paev@iict.bas.bg