deberta-base-75k — BabyLM 2026 strict-small (random-init control)
DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure),
75k byte-level BPE tokenizer, standard random embedding initialization —
the matched control for
deberta-base-75k-sam:
identical data, architecture, tokenizer, and schedule; the only
difference is embedding initialization. Training: lr 2e-4, batch 256,
grad-acc 4, MLM 15%.
Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM
2024 submission (Edman et al. 2024, "Are BabyLMs Second Language
Learners?") — a mixture
of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et
al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia,
Gutenberg, Switchboard). Within the strict-small 10M-word budget.
Intermediate checkpoints: chck_1M … chck_100M and step1000 …
step25740; main = final.
Code and analyses: https://github.com/bylinina/augustinian_babylm