quipu-moe-1B-A149M-chat-distill

A mixture-of-experts language model trained from scratch: 998,007,552 parameters in total, 148,660,992 active per token (998.0M total / 148.7M active). Own tokenizer, own data pipeline, no fine-tuned base. Code and ten languages: English, Indonesian, Malay, Simplified and Traditional Chinese, Japanese, Korean, Tamil, Hindi (Devanagari and romanised) and romanised Urdu.

This is the distilled chat model: quipu-moe-1B-A149M after the same short supervised fine-tune as quipu-moe-1B-A149M-chat, with one change: 3,100 coding answers from an open-weight teacher (Qwen/Qwen2.5-Coder-1.5B-Instruct, Apache-2.0) added to the human-written conversations. Every teacher answer kept was run against its unit tests in a sandbox; answers that failed, did not finish, or had no code were dropped (3,277 of 5,000 passed).

Both measured with the same harness, prompts and greedy decoding (see Evaluation). That is still low in absolute terms: reference models of similar total size score far higher (table below). At this size it is often wrong. It is published as an honest record of what a small sparse model learns when pretrained under a $20 compute cap.

Project page: https://quipulm.dev · Code: https://github.com/Aneek1/quipu

Architecture

Data

Mix achieved (share of training tokens): code 60.0%, eng_Latn 28.0%, other_languages 12.0%.

Training tokens in the shard set: 8,300,000,000 (the train split; validation and evaluation splits are separate).

Code licences kept: mit, apache-2.0, bsd-2-clause, bsd-3-clause, isc, cc0-1.0, unlicense.

Language identification: every text document was checked with the fastText specialist from AneekC/lid-specialists-9plus1 (MIT, revision 117966eb565a), the owner's own model; a document whose top label is not its source language was dropped, and FineWeb-2's cmn_Hani was split into Simplified and Traditional Chinese by its labels.

Benchmark decontamination

Documents dropped because they contain a HumanEval or MBPP problem (rule, from the shard manifest: a document is dropped when it contains a problem's text (HumanEval: prompt, canonical_solution; MBPP: code, task text) as a whitespace-collapsed substring of at least 60 characters, or, for one of the problem's n-gram texts (its solution; HumanEval also its prompt and each docstring in it), at least 50% and at least 2 of that text's distinct 13-grams (tokens: \w+ runs and single punctuation characters; a text with fewer than 2 13-grams takes no part)):

Chat fine-tuning data

Human-written conversations plus execution-verified teacher answers; from the SFT data manifest:

The distill source. 5,000 function-style problems (4,000 from nvidia/OpenCodeInstruct, rows with average_test_score == 1.0; 1,000 from codeparrot/apps), decontaminated against HumanEval and MBPP. The teacher, Qwen/Qwen2.5-Coder-1.5B-Instruct at revision 2e1fd397ee46, answered each once (greedy). An answer was kept only if it finished on its own, had a code block defining the expected function, and passed every assert in a sandbox: 3,277 kept (65.5%; OpenCodeInstruct 74.2%, APPS 31.0%). Reference solutions were never shown to the teacher or written to the training data. 3,100 kept answers are in the train split after decontamination; the held-out distill split is below.

Training

Pretraining (the base model, unchanged):

Chat fine-tune (this model):

A/B tests (run before the full run, results kept either way)

Runs (from results/ab/summary.md, verbatim)

Decisions (from results/ab/summary.md, verbatim)

Rules (on the decision loss above): the other option needs lower final loss by more than the seed noise; AttnRes <= 10% tokens/s cost; SiTU-GLU no more spikes (step loss > 1.5 x EMA, beta 0.98); FP8 >= 1.2x tokens/s, loss within noise, no more spikes. A missing arm keeps the simpler option (AdamW, no AttnRes, SwiGLU, bf16).

FP8

FP8 training was not tried in the paid run: the owner decided (2026-09-29) not to spend A/B time on the FP8 pair, so the run is bf16. The FP8 option exists in the training code (quipu/fp8.py) and was only checked on the laptop.

Evaluation

Bits per byte per language (base model, during pretraining)

Bits per byte (lower is better; comparable across tokenizers) on held-out text per language and held-out code. Definition: summed next-token loss in bits over the UTF-8 bytes of the target tokens. Every <|endoftext|> document separator is a target too: its loss counts but it is 0 bytes of text, so these numbers are slightly higher (more pessimistic) than a separator-free measure.

Code: HumanEval and MBPP

Published numbers are copied from their sources with the setting each source states; they were produced with other prompts, decoding and harnesses, so compare them with care (a row's setting says what differs).

Protocol (both quipu rows; results/code_eval_local/quipu-moe-1B-A149M-chat-distill.json and quipu-moe-1B-A149M-chat.json, run on the same laptop)

• Mode: chat.
• HumanEval prompt: chat template <|user|>...<|end|><|assistant|>; the user turn asks to complete the function and shows the published prompt in a ```python fence; the reply's first code block (or the whole reply) is the code.
• HumanEval stop sequences: none.
• MBPP prompt: chat template; the user turn is the task text, 'Your code should pass this test:' and the first assert; the reply's first code block is the code.
• MBPP stop sequences: none.
• Stop tokens: <|end|>, <|endoftext|>; at most 384 new tokens.
• Greedy: temperature 0 (argmax), one completion per problem -> greedy pass@1.
• Sampled: not run (greedy only).
• Execution: 10.0 s timeout per program; subprocess python -I in a temporary directory with a reliability guard (no deleting / renaming files, no writing outside the temp dir, no subprocesses, sockets cannot connect, resource limits on POSIX); best effort, not a security boundary: run it only where executing untrusted model output is acceptable.

Held-out chat loss

Held-out chat loss per source (the SFT's validation conversations, micro-batch 1), this model vs the chat model without distillation:

Expert usage and fixed-prompt chat samples were not re-measured for this model; the base and chat model cards have them for the models they describe. int4 was measured for this model (16 Oct 2026); see below.

Standalone-loader parity (max |logit difference| against the training code, fp32, real validation tokens): 2.6e-05, with logits up to about 22 and the top token agreeing at every position. The export check's tolerance was 5e-05 for this model (1e-05 for the others).

int4

Weight-only int4 (int4-groupwise-asym-v1, groups of 128 along the last dimension, fp16 scale + minimum per group) for the attention, shared and routed expert matrices; embedding bf16; router, norms and biases fp32. Dequantized to fp32 on load (the file is smaller; memory once loaded is not).

The first tokens of each evaluation split (count per row). bf16 = the weights rounded to bf16, arithmetic in fp32 (spec 12's baseline); fp32 = the trained weights as saved.

Standalone-loader parity (max |logit difference| against the training code): final (fp32) 2.6e-05; int4 (dequantized) 2.0e-05.

Usage

pip install torch safetensors tokenizers huggingface_hub

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("AneekC/quipu-moe-1B-A149M-chat-distill", "modeling_quipu_moe.py")
spec = importlib.util.spec_from_file_location("modeling_quipu_moe", path)
mq = importlib.util.module_from_spec(spec)
sys.modules["modeling_quipu_moe"] = mq
spec.loader.exec_module(mq)

model = mq.load("AneekC/quipu-moe-1B-A149M-chat-distill")                                    # fp32
# model = mq.load("AneekC/quipu-moe-1B-A149M-chat-distill", weights="model-int4.safetensors")  # int4 file
tok = mq.load_tokenizer("AneekC/quipu-moe-1B-A149M-chat-distill")
print(mq.chat(model, tok, [{"role": "user", "content": "Write a Python function that reverses a string."}]))

This is not a transformers model; there is no AutoModel class for it. The loader runs the experts one after another without a KV cache: correct, not fast.

Limitations

• Small: 148,660,992 parameters are active per token. It states false things with confidence; do not use its output as information.
• The nine languages other than English together are 12.0% of the training tokens (English 28.0%, code 60.0%; shard manifest, mix achieved). Expect weaker output in them than in English, and some mixing of languages.
• Code: greedy decoding solves 0.6% of HumanEval and 7.4% of MBPP problems (see above). Treat generated code as untested.
• The distillation data is Python, function-style and fairly short; gains outside that kind of task were not measured.
• Trained on web text and public code, which carry their biases; no safety tuning.
• How well it follows instructions was not measured beyond the held-out chat loss per source and the chat-mode HumanEval / MBPP numbers above.

Credits and licence

Weights and code: Apache-2.0. Language identification of the training text used AneekC/lid-specialists-9plus1 (MIT). Training data: FineWeb-Edu and FineWeb-2 (ODC-By 1.0, © Hugging Face) and the permissively licensed subset of codeparrot/github-code-clean (files keep their licences). Chat fine-tuning data: CohereForAI/aya_dataset (Apache-2.0), OpenAssistant/oasst2 (Apache-2.0), the stepbuild train split (FILE-block replies mined from permissively licensed public repositories, each keeping its repository's licence; built for this project, its test split and benchmark apps held out), and execution-verified answers from Qwen/Qwen2.5-Coder-1.5B-Instruct (Apache-2.0) to problems from nvidia/OpenCodeInstruct (CC-BY-4.0) and codeparrot/apps (MIT). HumanEval (MIT) and MBPP (CC-BY-4.0) were used for evaluation only and filtered (substring / 13-gram matching) from the training data.