polish-roberta-base-8k

This is a small model trained using knowledge distillation from polish-roberta-8k. For distillation, we used a combination of several loss functions: KL divergence for the MLM head, as well as MSE and cosine loss to align the last-layer representations of the teacher and the student. The model was trained for one epoch on the Polish subset of the FineWeb 2 dataset. We prepared a packed version of the dataset by concatenating documents into sequences of exactly 8192 tokens. We then trained the model using batches of 32 such sequences and a polynomial learning rate scheduler with a warmup of 1000 iterations and a maximum learning rate of 5e-5.

Evaluation

Evaluation shows that the model compares favorably to other base-sized encoder models (in the 100M-300M parameter range). We conducted fine-tuning experiments on 25 Polish tasks, the same set as for polish-roberta-8k. The table below presents detailed results compared to other popular models: herbert-base-cased and polish-roberta-base-v2.

Table 1. Comparison of the mean scores from five fine-tuning runs on 25 discriminative tasks in Polish. The evaluation metrics vary across tasks — the metric used for each task is specified in the METRIC column.

In addition to the detailed results for the three models shown above, we also present a summary of the evaluation of multilingual models that support Polish.

Table 2. Comparison of Polish and multilingual models.

Efficiency

The model includes a custom implementation supporting unpadding and sequence packing, which can significantly speed up inference or training while reducing memory consumption (more information here). Using this feature requires Flash Attention and the Transformers library version 5.4 or newer. To use unpadding, initialize the model with the trust_remote_code=true and attn_implementation="flash_attention_2" parameters, along with 16-bit precision.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "sdadas/polish-roberta-base-8k",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
    dtype=torch.bfloat16,
    device_map="cuda"
)
print(model.__class__.__name__)  # UnpadRobertaModel

Citation

@misc{dadas2026longcontext,
  title={Long-Context Encoder Models for Polish Language Understanding},
  author={Sławomir Dadas and Rafał Poświata and Marek Kozłowski and Małgorzata Grębowiec and Michał Perełkiewicz and Paweł Klimiuk and Przemysław Boruta},
  year={2026},
  eprint={2603.12191},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2603.12191}
}