ChonkyKedi-mini-base

ChonkyKedi-mini-base is a small dense Llama-style language model trained from scratch. It has 72.6M parameters (51.6M excluding the embedding table) and was pretrained on a 4.13B-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineWeb, Wikipedia (English, 2023-11-01), FineMath 4+, github-code-clean, SmolTalk).

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The ChonkyKedi family

ChonkyKedi models share one architecture, tokenizer and data pipeline, and differ only in scale. Each is trained on about 80 tokens per non-embedding parameter. The table is generated from the current presets.

Architecture

A dense, Llama-style decoder-only transformer (llama_gqa).

Training

Every training document was checked against every item of the five evaluation sets below (10-gram overlap, plus exact matches of short answers) and dropped on a hit. The validation set comes from shards the training stream never reads.

Evaluation

Validation loss over training: 5.07 (0.03B tok) → 3.53 (0.62B tok) → 3.47 (1.21B tok) → 3.44 (1.80B tok) → 3.42 (2.39B tok) → 3.41 (2.98B tok) → 3.39 (3.57B tok) → 3.13 (4.13B tok)

Zero-shot benchmarks (main.py --mode bench)

Each answer choice is scored by the log-probability of its text after the prompt (BLiMP: the whole sentence of each minimal pair). acc_norm divides by the answer's length in UTF-8 bytes; acc doesn't. Every item of each set is scored, not a sample; ± is one standard error.

Compared with other small models

ChonkyKedi columns are measured with this repo's evaluator on the checkpoints named; validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts, splits and length normalisation differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the first ChonkyKedi trained in the cloud: a dense 72.6M-parameter model (51.6M excluding the embedding table) that reached a validation loss of 3.129 after 4.13B tokens. It writes fluent, well-organised English in the register of textbooks, encyclopedias and children's stories, and it gets the broad frame of common topics right more often than not. But it's a model of the form of text far more than its content: most specific facts, dates and numbers it writes are wrong. It's a research and educational artifact, not a source of information.

What it does

• ✅ Writes fluent, grammatical paragraphs that stay on topic for several sentences. Grammar is its strongest skill: 74.5% on BLiMP across all 67,000 minimal pairs.
• ✅ Tells simple children's stories in the style of its synthetic-textbook data: "Once upon a time, a little cat" → "named Kira lived in a cozy burrow… One sunny afternoon, while playing near the fireplace, Kira met a wise old owl named Oliver."
• ✅ Gets the outline of well-known topics right: "The heart pumps" → "blood to the brain" or "blood to the muscles and other tissues"; "Photosynthesis is" → "a process in which plants convert light energy into chemical energy"; "Albert Einstein was" → "the father of modern physics… developed the theory of relativity".
• ✅ Gets a few very common facts right, often: "The capital of France is" → "Paris" (greedy and two of three sampled seeds); "Water boils at" → "100°C" (greedy only; sampled seeds gave 125 °C and 35 degrees).
• ✅ Reproduces document structure from its data mix: markdown headings and bullet lists, worksheet-style maths pages ("##### Stage: 2 Challenge Level:"), and code-shaped lines.
• ✅ Knows the chat format. About 4% of its pretraining data was SmolTalk conversations, so given <|user|>…<|assistant|> it replies in the assistant's voice, on topic, as a base model rather than an instruction-tuned one.
• ✅ Zero-shot benchmarks (acc_norm, every item scored): SciQ 67.8%, ARC-Easy 43.6%, PIQA 60.6%, HellaSwag 29.8%. That's level with or above GPT-2 (124M parameters) on all three tasks they share, from 4.13B training tokens, though prompts and normalisation differ between published numbers and ours.

What it can't do

• ❌ Facts. Most specific claims are invented: "The largest planet in the solar system is" → "the Sun" (greedy, and two of three seeds), "the Earth" or a planet "58,000 kilometers away from the Sun"; "The chemical symbol for gold is" → "gold" every time; "The French Revolution began in" → "1799" (greedy) and "September 1794"; "Albert Einstein was" → "born in 1882 in Vienna". It got 1789 once in four tries. Treat every name, date and number as made up.
• ❌ Arithmetic. "2 + 2 =" → "3" every time, followed by lists like "3 + 3 = 4, 4 + 4 = 5"; "7 + 6 =" → 10, 12, 16 or 8. Word problems fare no better: a train going 60 miles in 2 hours does "100 miles per hour", "125 miles per hour" or "35 miles per hour".
• ❌ Code. It writes code-shaped text but nothing that runs: def fibonacci(n): → f(n) = fib(n - 1) * fib(n - 1) * fib(n + 1)… or a line of zeros.
• ❌ Instructions. It's a base model with no instruction tuning. In the chat format it answers fluently and confidently but wrongly ("The largest planet in the solar system is the Sun, which is approximately 100 light-years from Earth"), and it ignores the task: asked for a haiku about the sea, it writes prose about the sea.
• ❌ Causal reasoning and history. "The main causes of the First World War were" → "the lack of food, the lack of water and the lack of air".
• ❌ Long-range coherence. It drifts and repeats within a paragraph, and loops on its own sentences when decoding is greedy.
• ❌ Long context. It was trained on 1,024-token sequences. YaRN scaling to 2,048 works (loss 2.65 with YaRN against 3.23 without on the 2× check), but nothing longer was tested.

Decoding

Greedy decoding falls straight into loops ("The symbol for gold is gold. The symbol for gold is gold…"; "It is the seat of the French people. It is the seat of the French people…"). Use sampling with temperature ≈ 0.7 and top-k ≈ 40, the defaults of ChonkyKediModel.generate, and expect to try a few seeds.

Usage

PyTorch (needs the chonkykedi package from the training code):

import torch
from chonkykedi.models.model import ChonkyKediModel
from chonkykedi.tokenizer.tokenizer import ChonkyTokenizer

model = ChonkyKediModel.from_pretrained(".")  # this folder: config.json + model.safetensors
tok = ChonkyTokenizer.load("tokenizer.json")
ids = torch.tensor([tok.encode("Photosynthesis is", add_bos=True)])
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Files

Acknowledgements

• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineWeb (ODC-BY 1.0).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-11-02 by prepare_release.py from llama_gqa_mini_4b.pt.