Chinese BabyLM - DeBERTa-V2 (14M)

Challenge: 2026 Chinese BabyLM Challenge

Model Description

DeBERTa-v2 masked LM (~14.67M params, 12L/256d/8h) with a custom BertTokenizer (WordPiece, vocab 12,800). Trained from scratch for 30 epochs on the official 100M Chinese BabyLM corpus using a physical 4-stage Curriculum Learning strategy.

Model Architecture & Configuration

This model is designed to squeeze the maximum cognitive capacity out of a small parameter budget:

• Architecture: DeBERTa-v2 (Masked Language Modeling)
• Total Parameters: 14,674,688 (~14.67M)
• Number of Layers: 12
• Hidden Size: 256
• Attention Heads: 8
• Max Sequence Length: 512

Tokenizer

• Tokenizer Class: BertTokenizer
• Algorithm: WordPiece
• Vocab Size: 12,800
• Note: Manual modification was performed on the trained tokenizer to ensure correct tokenization of Chinese punctuation (e.g., ，).

Training Data & Strategy

• Hardware: NVIDIA RTX 5090 GPU (used for both training and evaluation).
• Dataset: Official Chinese BabyLM corpus (102M).
• Curriculum Learning: We computed the exact token length for every sequence and physically split the dataset into four progressive cognitive levels:
  1. Easy (Length <= 32): Trained for 2 Epochs.
  2. Medium (Length <= 64): Trained for 3 Epochs.
  3. Hard (Length <= 128): Trained for 5 Epochs.
  4. Full Contexts (Max Length 512): Final sprint trained for 20 Epochs.
• Hyperparameters:
  • Total Epochs: 30
  • Peak Learning Rate: 5e-4
  • Batch Size: 32 (with Gradient Accumulation steps = 4)
  • Precision: Full Precision (FP32)
  • Optimizer: AdamW (weight_decay=0.01)

Evaluation Results

Below are the official benchmark results for this 14.67M model:

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

# Load the custom WordPiece tokenizer
tokenizer = AutoTokenizer.from_pretrained("LuRr/babylm-chinese-deberta-v2-14m-CL")

# Load the DeBERTa-v2 model
model = AutoModelForMaskedLM.from_pretrained("LuRr/babylm-chinese-deberta-v2-14m-CL")