HalleluBERT: Let every token that has meaning bear its weight

HalleluBERT is a family of RoBERTa-based Modern Hebrew language models pre-trained from scratch on ~49.1 GB of deduplicated Hebrew web text (HeDC4 / HeRo corpus) and Hebrew Wikipedia. The models aim to provide the first fully converged Hebrew RoBERTa encoder family, including a large variant, and to push state-of-the-art performance on core Hebrew benchmarks.

We release two variants:

HalleluBERT-base: 126M parameters (fp32)
HalleluBERT-large: 357M parameters (fp32)

Model Details

Detail	HalleluBERT-base	HalleluBERT-large
Architecture	RoBERTa-base	RoBERTa-large
Parameters	~126M	~357M
Tokenizer	GPT-2 style byte-level BPE (52,009 vocab)	Same
Pretraining corpus	HeDC4 (mC4 + OSCAR22) + Hebrew Wikipedia (~49.1 GB)	Same
Objective	Masked Language Modeling	Same
Training steps	100k updates, global batch size 8k	Same
LR schedule	10k warmup + polynomial decay	Same
Peak learning rate	0.0004	0.00015
Training time	~30.2 hours (TPUv4-128 pod)	~6.0 days (TPUv4-128 pod)
Precision	fp32	fp32
Framework	fairseq	fairseq

Downstream Evaluation

We evaluate HalleluBERT on three Hebrew benchmarks (following the HeRo suite, restricted to NER + sentiment):

NER (BMC split 1): micro-F1
NER (NEMO², token-level): micro-F1
Sentiment (SMCD, deduplicated): macro-F1

We select the best configuration by validation performance and report the best score out of 10 runs on the official test sets.

🧪 Evaluation Results

Legend: Bold = best, underline = second-best within each model size group.

Model	BMC (micro-F1)	NEMO (micro-F1)	AVG NER	SMCD (macro-F1)	AVG (all)
Large models
HalleluBERT_large	93.23	88.70	90.96	84.91	88.95
XLM-RoBERTa_large	92.31	86.41	89.36	83.74	87.49
Base models
HeBERT	89.33	76.16	82.74	82.64	82.71
AlephBERT	91.36	81.52	86.44	83.66	85.51
HeRo	92.00	83.35	87.68	80.95	85.43
HalleluBERT_base	93.33	87.06	90.20	83.09	87.83
mmBERT_small	83.96	71.95	77.96	81.89	79.27
AlephBERT-Gimmel	92.46	85.86	89.16	82.66	86.99
XLM-RoBERTa_base	86.32	79.37	82.84	82.07	82.59
mmBERT_base	84.61	77.97	81.29	83.55	82.04

Fairseq Checkpoint

Get the fairseq checkpoint here.

Citation

If you use HalleluBERT in your research, please cite the corresponding paper (replace with your final bib entry if you already have one):

@inproceedings{schmitt-2026-hallelubert,
  title = {HalleluBERT: Let Every Token That Has Meaning Bear Its Weight},
  author = {Schmitt, Raphael},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {3022--3030},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/3qdqexx4e9i2}
}

📜 License

MIT License

HalleluBERT/HalleluBERT_base