Chinese BabyLM 2026 submission (v3)

A Chinese masked language model for the NLPCC 2026 Chinese BabyLM Challenge, pretrained within the ≤102M Jieba-word data budget.

• Architecture: DeBERTa-v2 base (12 layers, hidden 768, 12 heads), ~102M params
• Tokenizer: character-level
• Objective: masked language modeling

Methodology details will be released in the technical report after the evaluation period.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("zymonody/chinese-babylm-v3")
model = AutoModelForMaskedLM.from_pretrained("zymonody/chinese-babylm-v3")