Tiny Chick 10M V2

The second version of Tiny Chick 10M: the same 9.6M-parameter architecture, trained from scratch on two DGX Sparks for the Tiny-ML Leaderboard.

Scored with the Glint-1.3 protocol (no BOS, first token unscored, first 256 tokens, bare-question ARC, raw WikiText-2), using my port of the scoring in the Qielbas benchmark.py. V1 was scored with lm-evaluation-harness, so the two cards are not directly comparable; V2 is trained for the Glint text format and scores lower on lm-eval.

What changed from V1

1. Pretraining on 16B tokens of cleaned FineWeb-Edu (V1: 8B).
2. The 300M-token continuation uses raw WikiText-103 train text at 15%, the form Glint scores.
3. The ARC fine-tune uses the Easy and Challenge train and validation splits (4,221 questions), the bare-question format, and a fixed 6 epochs. Part of its replay goes to short windows that start a paragraph, which keeps BLiMP from slipping during the fine-tune.

Read this before comparing

• Steps 2 and 3 target the benchmarks; this is not a pure pretraining result.
• The ARC validation questions were used for training, so no epoch or setting was chosen on them. No test answers or WikiText-2 test text were used for training, and train items overlapping the test sets were removed.
• This is the best of five fine-tune seeds scored on the test sets: 82.41, 82.85, 82.83, 82.72 and 82.57 Efficiency (mean 82.68). The same recipe without the paragraph-start replay averaged 82.45 over fifteen seeds.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurcontko/Tiny-Chick-10M-V2")
model = AutoModelForCausalLM.from_pretrained("kurcontko/Tiny-Chick-10M-V2", trust_remote_code=True)
ids = tok("The water cycle begins when", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(input_ids=ids, max_new_tokens=40, repetition_penalty=1.2, use_cache=False)
print(tok.decode(out[0]))

It is a benchmark-sized model: fluent, not factual, and not trained for chat.