LittleKedi-tiny-base

LittleKedi-tiny-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 2.0M parameters in total, of which 1.5M are active per token (0.50M excluding the embedding table). It was pretrained on a 1,000M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, SmolTalk, github-code-clean).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for tiny, the standard tiny model: 1.5M of its 2.0M parameters (78%) are active per token. Its siblings are tiny-sparse and tiny-wide-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Evaluation

Validation loss over training: 5.60 (16M tok) → 4.17 (164M tok) → 4.08 (295M tok) → 4.04 (442M tok) → 4.02 (590M tok) → 4.01 (737M tok) → 3.96 (868M tok) → 3.87 (1,000M tok)

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 28.4% on 2,089 questions, +3.4 points against 25% chance (about 3.6 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

At 0.5M active non-embedding parameters this model has learned the form of English much better than its content. It's a research artifact for studying small MoE models, not a source of information.

What it does

• ✅ Writes grammatical, fluent sentences for a paragraph or two, in the register of its training data (educational web text and textbooks): "Photosynthesis is the process by which" → "…it is used to monitor and measure the concentration of nutrients."
• ✅ Stays roughly on topic: a prompt about the heart leads to blood vessels, the body and disease; a story opening leads to characters and a plot.
• ✅ Picks up document structure from its data mix: markdown headings and worked "Example 1:" sections after maths prompts, dialogue turns after User: / Assistant: text.
• ✅ Shows some benchmark signal above chance (SciQ 56.3%, ARC-Easy 32.7%, MMLU biology & medicine 28.4% in the cloze format), mostly from matching answers to a passage or recognising which option sounds right, not from recalling facts.

What it can't do

• ❌ Facts. It reliably names the right kind of thing but the wrong thing: "The French Revolution began in" → "1906…"; "Albert Einstein was" → "a co-director of the French Marxi Palm." Treat every name, date and number it writes as made up.
• ❌ Arithmetic. "2 + 2 =" → "3", "7 + 6 =" → "6".
• ❌ Code. Code was 2% of its training data; given a Python function signature it produces empty docstrings and whitespace.
• ❌ Instructions or questions. It's a base model: it continues text rather than answering it, and with only 0.5M non-embedding parameters an instruction-tuned version won't be a reliable assistant either.
• ❌ Long-range coherence. Topics drift after a few sentences, and invented names appear ("the Grovi Palace").

Decoding

Greedy or low-temperature decoding falls into loops almost immediately ("The process of the plant is formed by the plant. The process of the plant is formed by the plant…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15, which is what produced the examples above.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia (Apache 2.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).

Card generated 2026-10-09 by prepare_release.py from tiny.pt (step 15257).