instanton-hybrid-dialogue

A 40.2M-parameter GPT-BERT masked–causal hybrid trained entirely from

scratch on a dialogue-heavy remix of the BabyLM 2026 Strict corpus, within

the Strict-Small budget (≤10M words).

Part of the instanton family, submitted to the

BabyLM 2026 Challenge. It is the mixture-variant

counterpart to qyxu1994/instanton-hybrid,

which uses the official mixture; architecture and recipe are identical, so the

pair isolates the effect of the data mixture.

| | |

|---|---|

| Parameters | 40,170,648 |

| Architecture | GPT-BERT (8 layers, 512 hidden, 8 heads, 1706 FFN) |

| Vocabulary | 16,384 (byte-level BPE, trained only on the in-budget corpus) |

| Training data | Dialogue-heavy remix, 9,999,167 words |

| Word exposure | 10 epochs ≈ 99.8M words |

| Pretrained weights used | None — random init, per challenge rules |

Training data

Resampled from the official detoxified BabyLM 2026 Strict (100M) pool — no

external data — by drawing contiguous 100-line blocks without replacement

(seeded) until per-source word targets were met. The mixture upweights spoken

and conversational text (~6.2M words from OpenSubtitles, BNC, Switchboard and

CHILDES), with children's books and simple encyclopedia making up the rest.

The research question: under a fixed 10M-word budget, does upweighting

cognitively motivated, child-directed and conversational data improve

sample efficiency?

Training objective and recipe

50/50 masked/causal blend after

Charpentier & Samuel (2024),

reproduced in-process on a single GPU (half the micro-batches of each optimizer

step masked, half causal). LAMB, max lr 0.007, cosine with 1.6% warmup /

1.6% cooldown, weight decay 0.1, grad clip 2.0, z-loss 1e-4, mask ratio

0.30→0.15, sequence length 128→256→512, 16,384 tokens/step, seed 42, bf16.

~2 hours on one A100-80GB.

Checkpoints

Nineteen word-exposure revisions as git branches: chck_1M … chck_9M,

chck_10M, chck_20M … chck_100M, plus main. Load with revision=.

Usage

Requires trust_remote_code=True (vendored modelling wrapper) and

transformers 4.51.x; 5.x is not supported.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tok = AutoTokenizer.from_pretrained(
    "qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    "qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)

Score as a masked LM (pseudo-log-likelihood / mntp).

Evaluation

| Task | This model | instanton-hybrid (official mixture) |

|---|---|---|

| BLiMP | 70.76 | 72.13 |

| BLiMP supplement | 61.06 | 60.86 |

| EWoK | 52.90 | 50.15 |

| Entity tracking (filtered) | 21.93 | 19.87 |

| COMPS | 53.15 | 52.96 |

| Global-PIQA | 29.25 | 37.14 |

| (Super)GLUE | 67.64 | 67.58 |

| Reading | 7.14 | 6.35 |

| Age of acquisition | 0 (r = −0.16, p = 0.008) | 0 (r = −0.11) |

| Leaderboard NLP average | 50.96 | 51.53 |

| Leaderboard Overall | 40.43 | 40.78 |

The dialogue-heavy mixture wins most zero-shot columns, GLUE and reading, and

notably recovers the EWoK and entity-tracking ground the official-mixture

hybrid gives up — evidence that the architecture and the mixture compose rather

than compete. It nonetheless has the lower leaderboard Overall, because of the

Global-PIQA result below.

Known limitations

Global-PIQA is below chance and unexplained. This model scores 16.5% on the

4-choice parallel subset (chance 25%) and 42% on the 2-choice non-parallel

subset (chance 50%) — it systematically prefers the wrong solution. This is not

a scoring artifact: the same code reproduces the official leaderboard score

exactly for a sibling model. Nor is it the mixture alone: a causal GPT-2 trained

on the same dialogue-heavy corpus scores 38.12. It appears specific to this

model and we have no explanation for it. **Treat this model's physical-commonsense

behaviour as unreliable.**

Age of acquisition is significantly negative (r = −0.165, p = 0.008), the

most negative in the instanton family. Upweighting child-directed and

conversational text moved this metric in the wrong direction, which is a

negative result worth reporting: feeding a model more child-like text does not

make its acquisition trajectory more child-like.

Intended use

A research artifact for studying sample-efficient language acquisition and data

mixture effects. Trained on <10M words; not suitable for deployment. See the

qyxu1994/instanton-hybrid-dialogue