LittleKedi-medium-wide-sparse-instruct

LittleKedi-medium-wide-sparse-instruct is the instruction-tuned (chat) version of LittleKedi-medium-wide-sparse-base, a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 305.4M parameters in total, of which 31.7M are active per token (23.29M excluding the embedding table). It was pretrained on a 20,400M-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, Reddit comments, github-code-clean, SmolTalk, Wikipedia (English, 2023-11-01)).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the instruction-tuned (chat) model. It answers in a chat format and stops when it's done, but it's still a very small model: answers are fluent and on-topic but often wrong or vague, and it invents facts confidently. It's an educational and research artifact; don't use it for anything that matters.

Chat format

<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>

The model writes its reply and ends it with <|end|>. The system turn is optional. The GGUF files embed this template, and chat_template.jinja / tokenizer_config.json carry it for other tools.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for medium-wide-sparse, the wide-sparse medium model: 31.7M of its 305.4M parameters (10%) are active per token. Its siblings are medium and medium-sparse.

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Training

Fine-tuning (SFT)

Validation loss on 1,201 held-out conversations (assistant tokens only): 1.973 → 1.859.

Evaluation

Zero-shot benchmarks (evals.py)

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Biology & medicine (cloze): 28.6% on 2,094 questions, +3.6 points against 25% chance (about 3.8 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Base vs instruct

Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. SFT mainly teaches format, turn-taking and when to stop, so large changes in knowledge benchmarks aren't expected. Chance on the multiple-choice tasks is 25%.

Compared with its siblings and other small models

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the instruction-tuned sparsest medium model: medium-wide-sparse-base (23.3M active non-embedding parameters per token, 240 experts, 305M in total) fine-tuned for one epoch on ~24K conversations, most of them written by DeepSeek V4.1 Flash for medium's 2,048-token context. It's the largest LittleKedi chat model: it answers in fuller, better-organised paragraphs than moderate-wide-sparse-instruct, gets the gist of common-knowledge questions right more often, and writes the simplest Python functions correctly. It still makes up specific details freely, with the same confidence as the right ones: a research artifact for studying sparsity and instruction tuning at small scale, not a source of information.

Who it is

It answers as LittleKedi, a small cat from Istanbul who has read a lot ("kedi" is Turkish for cat), and says it's a small language model that makes mistakes: "Who are you?" → "I'm LittleKedi! I'm a little cat from Istanbul, and I've spent a lot of time curled up with books. I'm happy to chat, explain things or help you think something through." The persona stays out of ordinary answers.

What it does

• ✅ Replies in the chat format and stops cleanly; greets back ("Hi!" → "Hello! I'm LittleKedi. How can I help you today?", 3 of 3 seeds).
• ✅ Gets the core of common-knowledge questions right more often than not: "What causes the seasons on Earth?" → "the tilt of the Earth's axis… about 23.5 degrees" (3 of 3 seeds name the tilt); "What is photosynthesis?" → "the process by which plants convert light energy into chemical energy. It uses sunlight, carbon dioxide, and water to create glucose"; a black hole is "a region in the universe where gravity is so strong that nothing can escape it, even light itself".
• ✅ Gives sensible, well-structured lists for everyday advice (sleep: a consistent schedule, a relaxing environment, no caffeine, alcohol or screens before bed, relaxation techniques).
• ✅ Writes the very simplest functions correctly: asked for the largest number in a list, it returned return max(lst) on one seed of three.
• ✅ Shortens a wordy sentence sometimes: "The committee has made the determination that the proposal will not be moving forward at this point in time" → "The committee has decided that the proposal is not moving forward."

What it can't do

• ❌ Specific details reliably. The main fact can be right and everything around it invented: Paris is "in the southwestern part of the country… with a population of about 36 million"; Einstein is "an American mathematician" on one seed and "a British biochemist… best known for his work on the structure of DNA" on another; one seed answers "What is the capital of France?" with "Louis XIV."
• ❌ Arithmetic reliably: 38.3% on 300 held-out one- and two-digit sums, differences and products (greedy decoding), a little below moderate-wide-sparse-instruct's 43.7%; "7 + 6" got 10, 12 and 12.
• ❌ Working code beyond one-liners. It writes correctly shaped Python but solves only 3.0% of 500 held-out MBPP problems (moderate-instruct: 1.8%); other seeds wrapped the right idea in a wrong max(nums, key=lambda x: x[1]).
• ❌ Explaining mechanisms: how a volcano erupts or how a virus differs from a bacterium comes out as fluent, confident nonsense ("a virus is a tiny, single-stranded virus"; magma reaching "10,000 degrees Fahrenheit").
• ❌ Staying faithful to a given text: asked to simplify a sentence, it sometimes reverses its meaning ("The committee has decided to move ahead with the project") and then explains changes it didn't make.
• ❌ Counting what it was asked for: asked for three sleep tips, it gave five on two seeds of three.

Benchmarks: about where the base model was (SciQ 73.8% vs 72.2%, ARC-Easy 46.1% vs 47.1%, HellaSwag 35.6% vs 36.0%, PIQA 63.9% vs 64.7%, acc_norm; MMLU cloze 27.4% vs 27.8%): instruction tuning changed the format, not what it knows.

Decoding: temperature ≈ 0.7, top-p ≈ 0.9, repetition penalty ≈ 1.15 (the GGUF files carry these as defaults), and a modest maximum length. The context window is 2,048 tokens, prompt and reply together.

Usage

llama.cpp (GGUF, llama.cpp's built-in deepseek2 architecture; the chat template is embedded):

llama-cli -m LittleKedi-medium-wide-sparse-instruct-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15

Ollama (uses the bundled template and params files): ollama run hf.co/edededdy/LittleKedi-medium-wide-sparse-instruct:Q8_0

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.chat import END, render_prompt
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([render_prompt(tok, [{"role": "user", "content": "What causes the seasons?"}])])
out = model.generate(ids, 200, temperature=0.7, top_p=0.9, repetition_penalty=1.15, stop_ids={tok.token_to_id(END)})
print(tok.decode(out[0, ids.shape[1]:].tolist()).replace(END, "").strip())

Or in a terminal: python chat.py --ckpt <checkpoint>.pt.

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: Reddit comments (no license stated on the dataset card; the comments are Reddit users' content).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Chat data: see the SFT table above for its sources, each under its own license.

Card generated 2026-11-01 by prepare_release.py from medium-wide-sparse-sft.pt (step 311278).