HalleluBERT_base (ONNX)

This is an ONNX version of HalleluBERT/HalleluBERT_base. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js

See the pipeline documentation for fill-mask: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.FillMaskPipeline

---

HalleluBERT: Let every token that has meaning bear its weight

HalleluBERT is a family of RoBERTa-based Modern Hebrew language models pre-trained from scratch on ~49.1 GB of deduplicated Hebrew web text (HeDC4 / HeRo corpus) and Hebrew Wikipedia. The models aim to provide the first fully converged Hebrew RoBERTa encoder family, including a large variant, and to push state-of-the-art performance on core Hebrew benchmarks.

We release two variants:

• HalleluBERT-base: 126M parameters (fp32)
• HalleluBERT-large: 357M parameters (fp32)

Model Details

Downstream Evaluation

We evaluate HalleluBERT on three Hebrew benchmarks (following the HeRo suite, restricted to NER + sentiment):

• NER (BMC split 1): micro-F1
• NER (NEMO², token-level): micro-F1
• Sentiment (SMCD, deduplicated): macro-F1

We select the best configuration by validation performance and report the best score out of 10 runs on the official test sets.

🧪 Evaluation Results

Legend: Bold = best, underline = second-best within each model size group.

Fairseq Checkpoint

Get the fairseq checkpoint here.

Citation

If you use HalleluBERT in your research, please cite the corresponding paper (replace with your final bib entry if you already have one):

@misc{scheibleschmitt2025hallelubertlettokenmeaning,
      title={HalleluBERT: Let every token that has meaning bear its weight}, 
      author={Raphael Scheible-Schmitt},
      year={2025},
      eprint={2510.21372},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.21372}, 
}

📜 License

MIT License