Tiny Chick 10M V3

A 9.6M-parameter language model trained from scratch for the Tiny-ML Leaderboard. Same architecture as Tiny Chick 10M V2.1, trained longer, with many more synthetic science passages before the ARC fine-tuning and a grammar term in the fine-tuning loss.

Scored with the Glint-1.3 protocol using glint_eval.py from this repo, a port of the scoring in the Qielbas benchmark.py.

Recipe

1. Pretraining: 24B tokens of cleaned FineWeb-Edu, the last 5.6B restricted to documents with educational score 3 or higher. 14 layers, width 240, 6 heads (2 KV heads), ReLU² MLP, QK-norm, partial RoPE, 4k BPE vocabulary, context 512.
2. Continuation: 600M tokens of 45% FineWeb-Edu, 15% WikiText-103 train text, 40% synthetic passages.
3. ARC fine-tuning: ten listwise fine-tunes on ARC with 85% language-model replay and a small grammar term, from the same continuation checkpoint with different seeds, with and without ARC-Challenge.
4. Soup: the ten fine-tuned checkpoints averaged in fp32 (blend.py). That average is the released model.

Synthetic passages. 69k short passages (20M tokens) written by MiMo-V2.6-Flash and Qwen3.8-27B about the concepts behind the ARC-Easy and ARC-Challenge train and validation questions, the OpenBookQA core facts and the QASC facts: question-and-answer pairs, fact lists and short explanations for a school reader. Outputs that were truncated, formatted, repetitive, duplicated, or shared a 13-word sequence with the ARC test or validation questions and answers or the WikiText-2 test text were dropped. The test questions were never shown to the generators.

Grammar term. 60k minimal pairs made from natural sentences (WikiText-103, FineWeb-Edu) by one rule-based corruption each: an agreement, pronoun, preposition or verb-form swap, two words exchanged, or a function word removed or inserted. A margin loss (weight 0.1, 2 nats) asks the model to prefer the original. No BLiMP sentences, templates or word lists were used. It adds about one BLiMP point and leaves ARC unchanged.

What to keep in mind

• This is benchmark-directed training, not a pure pretraining result. Before the ARC fine-tuning the model scores BLiMP 79.6 / ARC 56.3 / byte-ppl 2.313 on the validation axis.
• Training used the ARC train and validation splits, for the fine-tuning and as seeds for the passages, plus OpenBookQA and QASC. No test answers or WikiText-2 test text were used. Design choices were made on the validation axis; each candidate was scored once on the test sets.
• I scored fourteen candidate soups on the test sets across V2, V2.1 and V3; this is the best of them, and the others ranged from 82.5 to 86.0.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurcontko/Tiny-Chick-10M-V3")
model = AutoModelForCausalLM.from_pretrained("kurcontko/Tiny-Chick-10M-V3", trust_remote_code=True)
ids = tok("The water cycle begins when", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(input_ids=ids, max_new_tokens=40, repetition_penalty=1.2, use_cache=False)
print(tok.decode(out[0]))

A benchmark-sized model: fluent, not factual, and not trained for chat.