LittleKedi-medium-wide-sparse-base

LittleKedi-medium-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 305.4M parameters in total, of which 31.7M are active per token (23.29M excluding the embedding table). It was pretrained on a 20,400M-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, Reddit comments, github-code-clean, SmolTalk, Wikipedia (English, 2023-11-01)).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for medium-wide-sparse, the wide-sparse medium model: 31.7M of its 305.4M parameters (10%) are active per token. Its siblings are medium and medium-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Evaluation

Validation loss over training: 5.05 (16M tok) → 2.78 (2,933M tok) → 2.75 (5,849M tok) → 2.74 (8,766M tok) → 2.73 (11,665M tok) → 2.73 (14,582M tok) → 2.66 (17,498M tok) → 2.34 (20,400M tok)

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 29.9% on 2,094 questions, +4.9 points against 25% chance (about 5.2 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

The sparsest medium model: about 23.3M active non-embedding parameters per token, spread over 240 narrow experts (305M parameters in total), trained on 20.4B tokens. It's the strongest LittleKedi so far: validation loss 2.340 (moderate-wide-sparse: 2.478; the validation sets differ slightly), and better than moderate-wide-sparse on every benchmark. The gains are about what ~2.9x the training compute should buy, not a jump in kind: it gets the most common facts right reliably now, but it's still a model of the form of text more than its content, and specific facts, numbers and code are usually wrong. A research artifact for studying sparsity in small MoE models, not a source of information.

Most of its quality arrived in the last 20% of training. Validation loss sat on a plateau of ~2.73 for most of the run and fell by ~0.39 during the learning-rate decay, about three times the usual cooldown gain, which suggests the peak learning rate (2e-3) was higher than it needed to be.

What it does

• ✅ Writes fluent, well-organised paragraphs in the register of textbooks, encyclopedias and educational web text, staying on topic for a paragraph or more.
• ✅ Gets the most common facts right reliably: "The French Revolution began in" → "1789", "World War II ended in" → "1945" and "The first person to walk on the Moon was" → "Neil Armstrong" (each 3 of 3 seeds; the Moon landing's "1969" on 2 of 3), "The capital of France is" → "Paris" (2 of 3), "Photosynthesis is the process by which" → "plants convert light energy into chemical energy" (2 of 3), "The heart pumps" → "blood through the body to supply oxygen and nutrients to all cells". Einstein is "born in 1879" on 2 of 3 seeds and "a German-born physicist" on the third.
• ✅ Occasionally writes correct short code: one seed of "def is_prime(n):" produced the standard trial-division loop up to int(n**0.5) + 1. It knows the shape of common functions (docstrings, base cases, recursion) even when the logic is wrong (below).
• ✅ The best benchmark scores in the family (zero-shot): SciQ 72.2% (acc_norm; 78.8% acc), ARC-Easy 47.1% (53.2% acc), HellaSwag 36.0%, PIQA 64.7%, ARC-Challenge 26.3%, ArithMark-3 37.8%, up from 68.6 / 45.2 / 32.8 / 62.0 / 25.4 / 36.2 for moderate-wide-sparse. On the Open SLM Leaderboard's Intelligence Index that's ~19.4, against ~15.9 for moderate.
• ✅ Above chance on MMLU in the cloze format: 27.8% overall, ~30% on the social sciences and "other" categories. STEM (25.9%) and the humanities (26.1%) are still at about chance, and the multiple-choice letter format is at chance too (24.5%).

What it can't do

• ❌ Less common facts. "The chemical symbol for gold is" → "Cu" or "Pt"; "The largest planet in the solar system is" → "Ganymede", "the Earth" or a comparison with Mercury; "Water boils at" → "100°C" on one seed, "100 degrees Fahrenheit" and "103.6 degrees Fahrenheit" on the others. Around a right answer the details are made up: Einstein "born in Budapest, Hungary" or "in St. Petersburg, Russia", France "ruled by King Louis XVI… a monarchy under Napoleon III". Right and wrong answers come with the same confidence.
• ❌ Arithmetic beyond the very simplest. "2 + 2 =" → "4" on 2 of 3 seeds ("5" on the third), but "7 + 6 =" → "14", "15" or a multiple-choice question whose marked answer is "45".
• ❌ Working code, most of the time. Its Fibonacci is wrong on every seed ("return 2 * fibonacci(n - 1) + fibonacci(n - 2)", or the triangle-number formula), and two of three is_prime completions are just "return n % 2 == 0".
• ❌ Instructions or questions. It's a base model: it continues text. Asked "What is the largest planet in the solar system?" in a User: / Assistant: exchange, it repeats the question, writes a quiz page, or continues a dialogue that names Mercury or Pluto.
• ❌ Long-range coherence. A paragraph holds together; beyond that it drifts and repeats.

Decoding

Greedy decoding still loops, though later than the smaller models ("The heart is a muscle that pumps blood throughout the body. It is a muscle that is made up of two parts… The heart is a muscle that is made up"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: Reddit comments (no license stated on the dataset card; the comments are Reddit users' content).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).

Card generated 2026-10-30 by prepare_release.py from medium-wide-sparse.pt (step 311278).