LittleKedi-small-wide-sparse-base

LittleKedi-small-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 43.9M parameters in total, of which 6.5M are active per token (4.41M excluding the embedding table). It was pretrained on a 4,200M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for small-wide-sparse, the wide-sparse small model: 6.5M of its 43.9M parameters (15%) are active per token. Its siblings are small and small-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Evaluation

Validation loss over training: 4.95 (16M tok) → 2.93 (623M tok) → 2.86 (1,212M tok) → 2.84 (1,819M tok) → 2.83 (2,409M tok) → 2.81 (3,015M tok) → 2.75 (3,605M tok) → 2.56 (4,200M tok)

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 27.2% on 2,089 questions, +2.2 points against 25% chance (about 2.4 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

The sparsest small model: about 4.4M active non-embedding parameters per token, spread over 160 narrow experts (44M parameters in total). It's the strongest LittleKedi so far: validation loss 2.564 (~1.00 bits per byte) after 4.2B tokens, 0.08 below mini-wide-sparse, and below it at every point of training (by a steady ~0.055 on the same batches before the learning-rate decay). That's a real step, but a modest one: it writes slightly more fluent text and gets a few more common facts right, and is still a model of the form of text far more than its content. It's a research artifact for studying sparsity in small MoE models, not a source of information.

What it does

• ✅ Writes fluent paragraphs in the register of textbooks and educational web text, with full, grammatical sentences and topic-appropriate vocabulary throughout.
• ✅ Gets the broad frame of well-known topics right somewhat more often: "The heart pumps" → "blood throughout the body"; "Photosynthesis is the process by which" → "plants convert light energy into chemical energy"; "The capital of France is" → "Paris". These are still hits among misses: other seeds give "the capital of France" (circular), "photosynthetic microbes convert carbon dioxide into oxygen" and "blood and oxygen through the nose".
• ✅ Reproduces document structure from its data mix: Q&A lists, markdown headings, numbered exercises and code-shaped text. It has seen the recursive Fibonacci pattern ("return fibonacci(n-1) + fibonacci(n-2)"), but without the base case.
• ✅ The best benchmark scores in the family so far (acc_norm, zero-shot): SciQ 62.5%, ARC-Easy 38.3%, HellaSwag 29.0%, up from 60.7 / 37.7 / 28.2 for mini-wide-sparse. HellaSwag's gain is the most reliable (10,042 questions, ±0.45); ARC-Easy's is within noise.
• ✅ MMLU is still at chance, as expected at this size: 26.2% in the cloze format, 25.7% with letters. No subject group stands out: biology & medicine is 27.2% (2.4 standard errors above chance, down from mini-wide-sparse's 28.7%, so treat neither as a real skill).

What it can't do

• ❌ Facts. Most specific claims are wrong: "Water boils at" → "20 degrees Celsius"; "The largest planet in the solar system is" → "Earth, which is 215 times bigger than the Sun"; "The chemical symbol for gold is" → "gold"; "Albert Einstein was" → "born in 1869" or "a professor of physics at the University of New South Wales"; "The French Revolution began in" → "1798", "1785", "1760". Treat every name, date and number it writes as made up.
• ❌ Arithmetic. "2 + 2 =" → "3", "10" or "6", then a list of equally wrong sums; "7 + 6 =" → "10" or "9".
• ❌ Code. It writes code-shaped lines, but nothing that works: the Fibonacci recursion above has no base case, and other seeds loop over range(2, n + 1) and return n // 2.
• ❌ Instructions or questions. It's a base model: it continues text. Asked "What is the largest planet in the solar system?" in a User: / Assistant: exchange, it answers with a science-fair announcement at NASA's JPL.
• ❌ Long-range coherence. It stays on topic for a paragraph, but drifts and contradicts itself within a few sentences ("Photosynthesis… convert sunlight into electricity and produce oxygen, which then releases carbon dioxide").

Decoding

Greedy decoding falls straight into loops ("The heart is the largest organ in the body and is responsible for the body's metabolism. It is the largest organ in the body…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia (Apache 2.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-16 by prepare_release.py from small-wide-sparse.pt (step 64085).