LittleKedi-mini-wide-sparse-base

LittleKedi-mini-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 36.4M parameters in total, of which 4.9M are active per token (3.33M excluding the embedding table). It was pretrained on a 3,200M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for mini-wide-sparse, the wide-sparse mini model: 4.9M of its 36.4M parameters (13%) are active per token. Its siblings are mini and mini-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Evaluation

Validation loss over training: 5.08 (16M tok) → 3.01 (475M tok) → 2.94 (934M tok) → 2.91 (1,393M tok) → 2.90 (1,835M tok) → 2.89 (2,294M tok) → 2.81 (2,753M tok) → 2.64 (3,200M tok)

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 28.7% on 2,089 questions, +3.7 points against 25% chance (about 3.9 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the sparsest mini model: about the same 3.3M active non-embedding parameters per token as mini, spread over 128 narrow experts (36M parameters in total). It's the strongest LittleKedi so far by a clear margin: validation loss 2.645 (~1.03 bits per byte) after 3.2B tokens. On the same data, it went below anything mini reached in 1.57B tokens after only ~0.4B. It writes noticeably better English than the tiny models and occasionally gets a common fact right, but it's still a model of the form of text far more than its content. It's a research artifact for studying sparsity in small MoE models, not a source of information.

What it does

• ✅ Writes fluent, well-formed paragraphs in the register of textbooks and educational web text, with longer stretches of on-topic prose than the tiny models.
• ✅ Gets the broad frame of well-known topics right more often: "Albert Einstein was" → "one of the most influential figures in modern physics… his theory of general relativity"; "The heart pumps" → "blood to the brain"; "The capital of France is" → "Paris". These are hits among misses: the same prompts with other seeds give "Les Présentures", "Saint-Louis" and "the heart out of the lungs". Don't rely on any of them.
• ✅ Reproduces document structure from its data mix: markdown headings, Q&A-forum threads ("### Reno / What is Newton's law?"), #### answer lines after maths, and code-shaped text (def, if/else, return).
• ✅ The best benchmark scores in the family so far (acc_norm, zero-shot): SciQ 60.7%, ARC-Easy 37.7%, HellaSwag 28.2%. MMLU biology & medicine 28.7% in the cloze format, about 3.9 standard errors above chance. ARC-Easy and SciQ rose steadily over training (ARC-Easy 32.8% at 250M tokens, 37.7% at 3.2B).
• ✅ Overall MMLU is still at chance (26.5% cloze, 25.0% letter): expected at this size.

What it can't do

• ❌ Facts. Most specific claims are wrong: "Water boils at" → "20 degrees Fahrenheit"; "The largest planet in the solar system is" → "a red dwarf star"; "The chemical symbol for gold is" → "blue"; "The French Revolution began in" → "1837". Treat every name, date and number it writes as made up.
• ❌ Arithmetic. "2 + 2 =" → "3", then a list of equally wrong sums; "7 + 6 =" → "10 + 2".
• ❌ Code. It writes code-shaped lines, but nothing that works: def fibonacci(n): → return n, then an unrelated function and a comment that "the above code is not able to solve the problem".
• ❌ Instructions or questions. It's a base model: it continues text. Given a User: / Assistant: exchange it keeps the format but answers with invented numbers ("the largest planet… is 27.7 m deep").
• ❌ Long-range coherence. It stays on topic for longer than tiny, but drifts and repeats within a few sentences ("Photosynthesis is the process by which light and water are produced").

Decoding

Greedy decoding falls straight into loops ("The heart is the heart, the heart is the heart…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia (Apache 2.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-15 by prepare_release.py from mini-wide-sparse.pt (step 48827).