Tiny Chick 10M

A 9.6M-parameter English language model I trained from scratch on two DGX Sparks for the Tiny-ML Leaderboard. The name comes from the Polish kurczątko, a little chick.

Scored 0-shot with lm-evaluation-harness (--tasks blimp,arc_easy,wikitext, context 512).

How it was trained

1. Pretraining on 8B tokens of cleaned FineWeb-Edu.
2. A 300M-token continuation with 15% WikiText-103 train text mixed in.
3. A fine-tune on the ARC-Easy and ARC-Challenge train splits, with FineWeb-Edu replay.

Steps 2 and 3 target the benchmarks, so this is not a pure pretraining result. Without them the model scores 75.69 / 40.70 / 2.395. I never trained on test or validation answers or on WikiText-2 test text, and I removed train items that overlap with the test sets. The checkpoint was picked on ARC-Easy validation.

Architecture

14 layers, width 240, 6 heads (2 KV heads), 4,096-token BPE vocabulary. Compared with a plain Llama block it uses ReLU² MLPs, QK-norm, partial RoPE, exclusive self-attention and zero-initialised output projections. The code is in tinylm_arch.py.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurcontko/Tiny-Chick-10M")
model = AutoModelForCausalLM.from_pretrained("kurcontko/Tiny-Chick-10M", trust_remote_code=True)
ids = tok("The water cycle begins when", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(input_ids=ids, max_new_tokens=40, repetition_penalty=1.2, use_cache=False)
print(tok.decode(out[0]))

It is a benchmark-sized model: fluent, not factual, and not trained for chat.