EMNLP BabyLM 2026 — Multilingual OLMo2
Submission for the BabyLM 2026 Challenge Multilingual Track (EMNLP 2026 Workshop).
Model Description
A trilingual causal language model (English / Dutch / Chinese) pretrained from scratch on developmentally plausible text data, using the OLMo2 architecture.
- Architecture: Olmo2ForCausalLM
- Parameters: ~15M (512h / 8L / 8H / 1376ff)
- Tokenizer: BPE, vocab=32000
- Sequence length: 512 tokens
- Languages: English, Dutch, Chinese (equal data mix)
- Data: BabyLM 2026 multilingual corpus (100M tokens total)
Results
Multilingual Track leaderboard (as of submission):
| Model | Multilingual Avg | EN Avg | NL Avg | ZH Avg |
|---|---|---|---|---|
| This model | ~45-47 | - | - | - |
| Best baseline (gpt2-en_nld_zho_equal) | 38.86 | 40.04 | 40.02 | 36.51 |
Selected task scores:
- MultiBLiMP: EN 84.7%, NL 92.4%
- ZhoBLiMP: 78.5%
- SIB-200: EN 81.0%, NL 77.5%, ZH 77.0%
- MNLI: EN 57.8%, NL 57.3%, ZH 57.1%
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("exusiaiw/emnlp-babylm-2026")
tokenizer = AutoTokenizer.from_pretrained("exusiaiw/emnlp-babylm-2026")
Citation
BabyLM 2026 Challenge — EMNLP 2026 Workshop, Budapest