ModernBERT-HU-4k

ModernBERT-HU-4k is a Hungarian encoder pretrained from scratch with the ModernBERT architecture, a purpose-built Hungarian BPE tokenizer, and a 4,096-token context window. It was introduced in Detecting Toxic Communication in Hungarian: Theory, Corpus, and Thread-Conditioned Classification (WOAH 2026) as the backbone for thread-level toxicity classification, where whole comment threads must fit in one forward pass. Prior Hungarian encoders such as huBERT stop at 512 tokens.

The fine-tuned toxicity classifier built on this encoder is ModernBERT-HU-4k-thread-toxic.

Model details

Tokenizer

The BPE tokenizer was trained from scratch on the pretraining corpus. Fertility (subword tokens per alphabetic word, measured on the 61M-word silver toxicity corpus with URLs, numerals and punctuation-only tokens excluded):

Hungarian morphological aggression markers (T/V address shifts, sarcastic diminutive suffixes, euphemised taboo forms) stay single tokens instead of being split across subwords.

Training data

About 4.4B tokens from five sources:

Training procedure

The ModernBERT pretraining recipe was followed, changing only the context length schedule:

1. Stage 1, 1k context: the full corpus, 1,024-token sequences, 1 epoch (~4.4B tokens, 219,621 steps).
2. Stage 2, 4k context: the conversational corpus repacked into 4,096-token sequences for continued MLM pretraining, 1 epoch.

All training ran on a single NVIDIA RTX 5070 Ti (16 GB).

Evaluation

HuLU benchmark

ModernBERT-HU-4k leads all compared models on HuWNLI and is within 6 to 8 points overall of Racka-4B (27x larger; .751 / .502 / .730) and PULI-LlumiX-Llama-3.1 Chat (54x larger), without instruction tuning. HuLU does not test long-context ability.

Masked-token prediction

2,000 held-out samples from conversational-pretrain, random masking:

Downstream: thread-level toxicity

Fine-tuned on the silver toxicity corpus, the model reaches macro F1 0.811 (face-aggression, T1) and 0.836 (norm violation, T3) on a 485-item human gold set, statistically tied with the best zero-shot frontier LLMs. See ModernBERT-HU-4k-thread-toxic.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "magyar-nlp-szine-java/ModernBERT-HU-4k"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo)

text = "A magyar [MASK] az egyik legszebb nyelv Európában."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

mask_idx = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
top5 = torch.topk(logits[0, mask_idx], 5).indices[0]
print([tokenizer.decode(t) for t in top5])

The encoder is meant for fine-tuning on Hungarian downstream tasks (classification, NER, sentence-pair tasks), especially where inputs are longer than 512 tokens.

Limitations

• Trained on Hungarian only; not suitable for other languages.
• One epoch over ~4.4B tokens; further pretraining would likely help.
• The pretraining data deliberately includes toxic and politically charged text. The model has not been audited for social bias, and its representations should not be assumed neutral.

Citation

Proceedings details (pages, anthology link) will be added once the WOAH 2026 proceedings are published.

@inproceedings{hatvani-madarasz-2026-detecting,
  title     = {Detecting Toxic Communication in {H}ungarian: Theory, Corpus, and Thread-Conditioned Classification},
  author    = {Hatvani, P{\'e}ter and Madar{\'a}sz, G{\'a}bor},
  booktitle = {Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH 2026)},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

License

Apache 2.0