BabyDeBERTa-24M (hanzi-aug v8s) — ChineseBabyLM 2026

Small from-scratch DeBERTa-v2 masked language model (~26M params) for the ChineseBabyLM 2026 all-track challenge. Trained only on the official chinese-babylm-org/babylm-zho-100M corpus (≤102M tokens) plus synthesized HANZI auxiliary data (pinyin homophone + character-structure facts), up-weighted during training via weighted sampling.

backend: mlm (this is a masked LM, evaluate with the MLM backend, not causal.)

Scores (full eval pipeline, step-030000)

HANZI pinyin	HANZI structure	ZhoBLiMP	afqmc	ocnli	tnews	cluewsc2020	word_fmri	fmri
98.30	83.70	75.67	69.14	67.93	54.08	64.14	55.86	8.57

Aggregates: HANZI 91.0 · NLU 66.2 · Cog 32.2 → Overall ≈ 63.1

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ireallydk/babydeberta-24m-hanzi-v8s")
model = AutoModelForMaskedLM.from_pretrained("ireallydk/babydeberta-24m-hanzi-v8s")

Method

12k WordPiece tokenizer (NFKC, force-covers structure components, built only from training data + general hanzi dictionaries — not the eval set). HANZI aux synthesized from general sources (char↔pinyin homophones; whole↔component structure from makemeahanzi + cjkvi-ids). Weighted sampling (aux_fraction=0.4). Anti-leakage: corpus selection never uses the eval set; exact eval test sentences removed.

Training data + full pipeline: https://github.com/yjchen-713705/chinese-babylm-2026 (branch babydeberta-24m-hanzi-v8s).

Loading…