LittleKedi-moderate-wide-sparse-base

LittleKedi-moderate-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 173.4M parameters in total, of which 19.4M are active per token (13.12M excluding the embedding table). It was pretrained on a 12,500M-token mix (FineWeb-Edu, cosmo-v2, wikipedia, FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for moderate-wide-sparse, the wide-sparse moderate model: 19.4M of its 173.4M parameters (11%) are active per token. Its siblings are moderate and moderate-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Evaluation

Validation loss over training: 5.04 (16M tok) → 2.89 (1,802M tok) → 2.86 (3,588M tok) → 2.84 (5,374M tok) → 2.84 (7,143M tok) → 2.83 (8,929M tok) → 2.76 (10,715M tok) → 2.48 (12,500M tok)

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 29.1% on 2,089 questions, +4.1 points against 25% chance (about 4.3 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

The sparsest moderate model: about 13.1M active non-embedding parameters per token, spread over 240 narrow experts (173M parameters in total), trained on 12.5B tokens. It's the strongest LittleKedi so far by a wide margin: validation loss 2.478 (~0.89 bits per byte, against ~1.00 for small-wide-sparse; the validation sets differ slightly, but on FineWeb-Edu alone it's ~12% better per byte). That's the biggest step in the family, and it shows: the first LittleKedi that gets some common facts right more often than not, and whose benchmark scores clearly separate from the smaller sizes. It's still a model of the form of text more than its content: specific facts, numbers and code are usually wrong. A research artifact for studying sparsity in small MoE models, not a source of information.

Its first two layers are dense (the preset's other layers are MoE). With only one, the first MoE layer's routing never balanced during training: its top expert choices were nearly tied, so load-balancing made it oscillate.

What it does

• ✅ Writes fluent, well-organised paragraphs in the register of textbooks, encyclopedias and educational web text, staying on topic for a paragraph or more.
• ✅ Gets a few common facts right reliably: "The French Revolution began in" → "1789" (3 of 3 seeds; every smaller LittleKedi missed it), "2 + 2 =" → "4" (3 of 3), "Photosynthesis is the process by which" → "plants convert light energy into chemical energy" (2 of 3), "The heart pumps" → "blood throughout the body, and it carries oxygen". Others are hits among misses: "Paris", Einstein as "a German physicist" or "born in 1879" and the theory of relativity, "Water boils at 100 degrees", each on about one seed in three.
• ✅ Has seen the shape of common code: the recursive Fibonacci ("return fibonacci(n-1) + fibonacci(n-2)"), docstrings, def main():, though the details are wrong (below).
• ✅ The best benchmark scores in the family (zero-shot): SciQ 68.6% (acc_norm; 76.0% acc), ARC-Easy 45.2% (50.4% acc), HellaSwag 32.8%, up from 62.5 / 38.3 / 29.0 for small-wide-sparse.
• ✅ The first LittleKedi clearly above chance on MMLU in the cloze format: 27.4% overall, ~29.4% on the social sciences and "other" categories. STEM is still at chance (25.5%), and the multiple-choice letter format is at chance too (26.0%): it isn't a knowledge benchmark this model can do much with yet.

What it can't do

• ❌ Specific facts. Most names, numbers and dates are still made up: "The chemical symbol for gold is" → "Cu" or "CuSO4"; "The largest planet in the solar system is" → "called Triton"; "Water boils at" → "102 degrees Fahrenheit"; Einstein "born on January 1, 1869 in St. Petersburg"; the French Revolution "ruled by King Louis XIV… under Napoleon III". Right and wrong answers come with the same confidence.
• ❌ Arithmetic beyond the very simplest. "2 + 2 = 4", but "7 + 6 =" → "12" or "9", and the sums it lists afterwards drift ("4 + 0 = 5").
• ❌ Working code. Its Fibonacci has the wrong base case ("if n == 0: return 1", none for 1) and comments with made-up outputs ("# prints 15").
• ❌ Instructions or questions. It's a base model: it continues text. Asked "What is the largest planet in the solar system?" in a User: / Assistant: exchange, it writes another turn of a classroom transcript.
• ❌ Long-range coherence. A paragraph holds together; beyond that it drifts and repeats.

Decoding

Greedy decoding still loops ("The heart is the largest organ in the human body… The heart is the largest muscle in the human body. It is responsible for pumping blood throughout the body. The heart is…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-19 by prepare_release.py from moderate-wide-sparse.pt (step 190733).