Sign up

qyxu1994/babylm-2026-strict-small-dialogue-heavy

🤗 On Hugging Facemit27M params109 MBsafetensors✓ HF checksums availableupdated today
Magnet

babylm-2026-strict-small-dialogue-heavy

BabyLM Challenge 2026 — Strict-Small track (≤10M words). Part of

*Reweighting Child-Directed and Conversational Data for Sample-Efficient

BabyLM Pretraining* (strict_small_dialogue_heavy).

  • Architecture: GPT-2-style decoder-only transformer, from-scratch init

(no pretrained weights): 6 layers, 512 hidden, 8 heads,

512-token context, 27.4M parameters (tied embeddings).

  • Objective: causal language modeling.
  • Tokenizer: byte-level BPE, 16k vocab, trained only on this run's

≤10M-word training mixture.

  • Training data: 9,999,167 words (custom mixture resampled from the official 100M Strict pool). Word counts use

whitespace tokenization matching the official BabyLM 2026 dataset cards.

Manifest and datasheet in the GitHub repo.

  • Training: 10 epochs (~100M words exposure), packed

512-token sequences, effective batch 128, lr 5e-4 cosine (3% warmup),

weight decay 0.1, bf16, seed 42, single A100-80GB.

Intermediate checkpoints

As required by BabyLM 2026, checkpoints at every 1M words seen (to 10M) and

every 10M (to 100M) are available as revisions chck_1M … chck_100M:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("qyxu1994/babylm-2026-strict-small-dialogue-heavy", revision="chck_5M")
tokenizer = AutoTokenizer.from_pretrained("qyxu1994/babylm-2026-strict-small-dialogue-heavy")

Evaluation

Evaluated with the official babylm-eval

pipeline (zero-shot: BLiMP, BLiMP-supplement, EWoK, entity tracking, COMPS,

reading; fine-tuning: BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, WSC):

./eval_zero_shot.sh qyxu1994/babylm-2026-strict-small-dialogue-heavy causal evaluation_data/full_eval

Mixture (strict_small_dialogue_heavy)

| Source | Words |

|---|---|

| open_subtitles | 3,999,885 |

| bnc_spoken | 1,999,986 |

| switchboard | 239,559 |

| childes | 2,259,971 |

| gutenberg | 749,839 |

| simple_wiki | 749,927 |

Limitations

Single pretraining seed; 27M parameters; English only; trained on ≤10M words —

a research artifact for sample-efficiency study, not a production model.

Citation

Paper forthcoming (BabyLM Challenge 2026). Please also cite the BabyLM 2026

call for papers (arXiv:2602.20092).