MiDo-70M v3

MiDo-70M is a tiny language model trained from scratch by UltraLabs (SmallAICreator). It has 70.4M parameters (45.2M without the embeddings) and was trained mostly by knowledge distillation: it learned from the full output distributions of SmolLM2-360M instead of only next-token labels.

v3 continues v1 with another 5.24B distilled tokens, so it has seen about 10.9B tokens in total, all on a single Kaggle TPU v5e-8 session per version.

Head-to-head with TII's Falcon-H1-Tiny-90M-Instruct (2026): MiDo beats it on trivia both models were trained on (82% vs 73%) and is slightly ahead on ARC-Easy (52.7 vs 50.8), with ~20M fewer parameters and ~1.4% of the training tokens (10.9B vs 800B). Details and caveats below.

Model details

Benchmarks

All scores use lm-evaluation-harness 0.4.13, zero-shot, fp32, max length 2048, and the full test sets. The Pythia-70M column comes from our own run with the same settings.

• MiDo v3 Chat beats Pythia-70M on 7 of 8 tasks with about 27× less training data (~11B tokens vs ~300B).
• v3's extra tokens mainly improved language modeling: BLiMP grammar went up almost 4 points and WikiText bits per byte dropped to 0.957. The six-task average stayed about the same as v1.
• For reference, SmolLM2-135M (twice the size, ~2T tokens) averages 49.3 on the same six tasks, and random guessing averages 29.2.

TruthfulQA (lm-eval truthfulqa_mc1 / truthfulqa_mc2):

Chat test. We asked 12 everyday questions, 3 samples each, with temperature 0.5, top-p 0.9 and repeat penalty 1.15. An answer only counted if it was correct and contained no made-up details.

MiDo keeps its answers short, so it invents fewer details. SmolLM2-135M-Instruct knows more trivia but adds confident made-up facts to longer answers.

Fact recall on 300 trained facts: the base answers 98.3% correctly. The chat model keeps about 87%: it answers in full sentences, and chat tuning made it forget about 12%.

vs Falcon-H1-Tiny-90M

Falcon-H1-Tiny-90M-Instruct is TII's 2026 tiny chat model: 90M parameters, ~800B training tokens (25% of them SFT data such as Tulu 3, repeated 100+ times), then DPO.

How the trivia test works:

• Every candidate question was checked against MiDo's training fact bank (30,970 facts) and against Tulu 3 SFT, a dataset TII names for Falcon's SFT data. Questions were then grouped by whether both models or neither model had seen them.
• Each set ran three times: greedy, plus two sampled runs (temperature 0.5, top-p 0.9, top-k 40, repeat penalty 1.15). Answers were keyword-graded, then checked by hand.

Notes:

• On ARC-Easy the gap is about 2 points, with ±1.0 standard error per score: MiDo is on par to slightly ahead.
• On facts neither model studied, Falcon knows more. At this size both models mostly answer from what they saw repeated in training.
• MiDo's efficiency comes from distillation. Its teacher, SmolLM2-360M, was trained on ~4T tokens, so each of MiDo's tokens carries a lot of the teacher's knowledge.

Use it

• On Android: open the file in L-AI. At 143 MB it runs fast on almost any phone.
• llama.cpp: llama-cli -m MiDo-70M-v3-Chat-F16.gguf -cnv
• Suggested sampling: temperature 0.5, top-p 0.9, repeat penalty 1.15.
• To use the base model, give it a plain prompt such as Q: What do bees make?\nA: instead of the chat format.

Limitations

• It's a 70M model. It makes up facts, gets multi-step math wrong and writes shaky code.
• Safety tuning is weak: it often follows harmful requests instead of refusing them. Don't rely on it for anything safety-critical.
• It mostly ignores custom system prompts.
• English only.

Credits and license

Made by SmallAICreator · UltraLabs. Released under Apache-2.0. Teacher model: SmolLM2-360M (Apache-2.0). Some fine-tuning data has its own license, such as PKU-SafeRLHF (CC BY-NC 4.0); check those if you plan commercial use.