Chinese BabyLM 2026 — DeBERTaV2 (v3)

Submission for the Chinese BabyLM 2026 Challenge (NLPCC 2026).

Model Description

A compact DeBERTaV2 masked language model pretrained from scratch on Chinese child-directed and child-produced text. Designed for data-constrained language modeling with a pure-Chinese WordPiece tokenizer.

Architecture: DeBERTaV2ForMaskedLM (disentangled attention, relative position)
Parameters: ~6M (256h / 4L / 4H / 1024ff)
Tokenizer: WordPiece, vocab=8000 (limit_alphabet=2000), NFKC + lowercase
Sequence length: 256 tokens
Training: 5000 steps, batch=128, lr=5e-4, bf16, dynamic span masking (mean_span=3.0)
Data: Full chinese-babylm-org/babylm-zho-100M corpus (183K documents)

Results

Best model from a 2x3 ablation study (architecture x radical embedding):

Model	NLU	HANZI	Cog	Overall
DeBERTaV2 (this model)	60.22	84.15	31.14	58.50
BERT plain	58.49	85.25	31.41	58.38
ModernBERT plain	59.61	80.88	31.34	57.28

Key findings:

Tokenizer granularity is the primary factor for HANZI track performance (8K vocab >> 32K vocab)
DeBERTaV2 achieves the highest NLU scores (zhoblimp: 69.89)
Radical embedding hurts performance with character-level tokenization

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("exusiaiw/chinese-babylm-2026-v3")
tokenizer = AutoTokenizer.from_pretrained("exusiaiw/chinese-babylm-2026-v3")

Citation

Chinese BabyLM 2026 Challenge — NLPCC 2026 Workshop

加载中…