ShallowSeek-mini-it-GGUF
GGUF builds of ShallowSeek-mini-it for llama.cpp, Ollama and LM Studio. The PyTorch weights (model.safetensors), config and tokenizer are in the main repo.
ShallowSeek-mini-it is the instruction-tuned (chat) version of ShallowSeek-mini-base, a tiny DeepSeek-V3-style Mixture-of-Experts model trained from scratch on a laptop CPU. It has 11.0M parameters, of which 6.3M are active per token.
Part of the ShallowSeek family. Not affiliated with DeepSeek.
It answers in a chat format and stops when it's done, but it's still an 11M-parameter model: answers are fluent and on-topic but often wrong or vague, and it invents facts confidently. It's an educational and research artifact; don't use it for anything that matters.
Chat format
<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>
The model writes its reply and ends it with <|end|>. The GGUF embeds this template, and the GGUF repo includes Ollama template and params files.
Fine-tuning (SFT)
| Base model | ShallowSeek-mini-base (pretrained on 286M FineWeb-Edu tokens) |
| Conversations | 16,347 (15,530 train / 817 validation), 1.62M assistant tokens trained on |
| Grounded Q&A (~36% of assistant tokens) | ~10.6K single-turn question/answer pairs generated by Qwen3.8-9B from FineWeb-Edu documents in ShallowSeek's own pretraining data, so answers concern text the model saw. About 2% of pairs still refer to "the document"; these were left in |
| General chat (~64%) | 5,684 conversations from HuggingFaceTB/smol-smoltalk: all 1,994 everyday-conversations (small talk, multi-turn), plus openhermes, smol-constraints and smol-magpie-ultra-short (capped at 15% of assistant tokens), filtered to remove code and to fit 1,024 tokens |
| Loss | On assistant replies and their closing `< |
| Schedule | 1942 steps, batch 16, 2 epochs, AdamW, LR 1.5e-4 → 1.5e-5 cosine (about 10% of the pretraining peak) |
| MoE routing | Load-balancing biases frozen at their pretrained values during SFT, with padding excluded from the expert-load statistics |
Validation loss on 817 held-out conversations (assistant tokens only): 3.779 → 2.949.
Evaluation: base vs IT
Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. Chance on MMLU is 25%.
| Metric | Base | IT |
|---|---|---|
| Probes top-5 | 28.6% | 35.7% |
| Mean answer log-prob | -5.68 | -5.40 |
| True-vs-false pairs won | 58.3% | 66.7% |
| MMLU (letter) | 24.9% | 25.6% |
| MMLU (cloze) | 25.5% | 25.6% |
| MMLU biology & medicine (cloze) | 27.8% | 27.7% |
SFT mainly teaches format, turn-taking and when to stop, and MMLU is essentially unchanged. The factual probes improved slightly, plausibly because the grounded Q&A restates facts from documents the model pretrained on. With 14 probes and 12 pairs, treat that as a modest effect rather than a large one.
Usage
Ollama (uses the bundled chat template):
ollama run hf.co/edededdy/ShallowSeek-mini-it-GGUF:Q8_0
llama.cpp:
llama-cli -m ShallowSeek-mini-it-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15
PyTorch: see chat.py in the training code:
python chat.py --ckpt mini-sft.pt --temperature 0.7 --rep-penalty 1.15
Files
| File | Contents |
|---|---|
model.safetensors |
PyTorch weights (including the MTP module), our own parameter names |
config.json |
ModelConfig for deepseek_moe |
tokenizer.json |
Byte-level BPE tokenizer (Hugging Face tokenizers format) |
chat_template.jinja |
Chat template (Jinja), the same one embedded in the GGUF |
tokenizer_config.json |
Special tokens (`< |
ShallowSeek-mini-it-f16.gguf |
GGUF (f16) for llama.cpp / Ollama / LM Studio |
ShallowSeek-mini-it-q8_0.gguf |
GGUF (q8_0) for llama.cpp / Ollama / LM Studio |
Acknowledgements
- Architecture after DeepSeek-V3 (DeepSeek-AI, 2024).
- Pretraining data: FineWeb-Edu (Hugging Face, ODC-BY 1.0).
- Chat data: smol-smoltalk (Hugging Face, Apache-2.0).
- Synthetic Q&A written by Qwen3.8-9B.
Card generated 2026-10-01 by prepare_release.py.