Tiny Chick 10M V2.1

A 9.6M-parameter language model trained from scratch for the Tiny-ML Leaderboard. It is Tiny Chick 10M V2 with one change: before the ARC fine-tuning, the model also reads a small set of synthetic science passages about the concepts behind the ARC training questions.

Scored with the Glint-1.3 protocol (no BOS, first token unscored, first 256 tokens, bare-question ARC, raw WikiText-2 test) using glint_eval.py from this repo, a port of the scoring in the Qielbas benchmark.py.

Recipe

1. Pretraining: 16B tokens of cleaned FineWeb-Edu. 14 layers, width 240, 6 heads (2 KV heads), ReLU² MLP, QK-norm, partial RoPE, 4k BPE vocabulary, context 512. Same checkpoint as V2.
2. Continuation: 300M tokens of 75% FineWeb-Edu, 15% raw WikiText-103 train text, 10% synthetic passages.
3. ARC fine-tuning: ten listwise fine-tunes on ARC with 85% language-model replay, started from the same continuation checkpoint with different seeds and two variants (with and without ARC-Challenge).
4. Soup: the ten fine-tuned checkpoints averaged in fp32 (blend.py). That average is the released model.

The synthetic passages

12,705 short expository passages (3.3M tokens) written by MiMo-V2.6-Flash. For every ARC-Easy and ARC-Challenge train and validation question, the generator (MIT-licensed) was shown the question with its correct answer and asked to write prose about the underlying concept from three angles: a textbook explanation, an everyday example, and the key terms plus the general rule. No quiz, no answer choices, no restating of the question. Passages sharing any 13-word sequence with the ARC test or validation questions or the WikiText-2 test text were removed. The test questions were never shown to the generator.

What to keep in mind

• This is benchmark-directed training, not a pure pretraining result. The checkpoint before the ARC fine-tuning scores BLiMP 80.1 / ARC 45.1 / byte-ppl 2.323 on the validation axis.
• Training used the ARC train and validation splits, both for the fine-tuning and as seeds for the synthetic passages. No test answers or WikiText-2 test text were used. Design choices were made on a separate protocol with the validation split held out.
• The synthetic passages are worth about +1.1 Efficiency over the same recipe without them (V2: 83.08), almost all of it on ARC-Easy. The same gain showed up on a second pretraining seed.
• I scored seven candidate soups on the test sets while developing V2 and V2.1; this is the best of them, and the others ranged from 82.5 to 83.6.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurcontko/Tiny-Chick-10M-V2.1")
model = AutoModelForCausalLM.from_pretrained("kurcontko/Tiny-Chick-10M-V2.1", trust_remote_code=True)
ids = tok("The water cycle begins when", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(input_ids=ids, max_new_tokens=40, repetition_penalty=1.2, use_cache=False)
print(tok.decode(out[0]))

A benchmark-sized model: fluent, not factual, and not trained for chat.