quipu-moe-1B-A149M

A mixture-of-experts language model trained from scratch: 998,007,552 parameters in total, 148,660,992 active per token (998.0M total / 148.7M active). Own tokenizer, own data pipeline, no fine-tuned base. Code and ten languages: English, Indonesian, Malay, Simplified and Traditional Chinese, Japanese, Korean, Tamil, Hindi (Devanagari and romanised) and romanised Urdu.

This is the base model: it continues text and code; it does not follow instructions (the chat model is AneekC/quipu-moe-1B-A149M-chat).

At this size it is often wrong. It is published as an honest record of what a small sparse model learns under a $20 compute cap, with every A/B result, including the ones that lost. Fields marked not yet measured had no result file when this card was generated.

Project page: https://quipulm.dev · Code: https://github.com/Aneek1/quipu

Architecture

Data

Mix achieved (share of training tokens): code 60.0%, eng_Latn 28.0%, other_languages 12.0%.

Training tokens in the shard set: 8,300,000,000 (the train split; validation and evaluation splits are separate).

Code licences kept: mit, apache-2.0, bsd-2-clause, bsd-3-clause, isc, cc0-1.0, unlicense.

Language identification: every text document was checked with the fastText specialist from AneekC/lid-specialists-9plus1 (MIT, revision 117966eb565a), the owner's own model; a document whose top label is not its source language was dropped, and FineWeb-2's cmn_Hani was split into Simplified and Traditional Chinese by its labels.

Benchmark decontamination

Documents dropped because they contain a HumanEval or MBPP problem (rule, from the shard manifest: a document is dropped when it contains a problem's text (HumanEval: prompt, canonical_solution; MBPP: code, task text) as a whitespace-collapsed substring of at least 60 characters, or, for one of the problem's n-gram texts (its solution; HumanEval also its prompt and each docstring in it), at least 50% and at least 2 of that text's distinct 13-grams (tokens: \w+ runs and single punctuation characters; a text with fewer than 2 13-grams takes no part)):

Training

A/B tests (run before the full run, results kept either way)

Runs (from results/ab/summary.md, verbatim)

Decisions (from results/ab/summary.md, verbatim)

Rules (on the decision loss above): the other option needs lower final loss by more than the seed noise; AttnRes <= 10% tokens/s cost; SiTU-GLU no more spikes (step loss > 1.5 x EMA, beta 0.98); FP8 >= 1.2x tokens/s, loss within noise, no more spikes. A missing arm keeps the simpler option (AdamW, no AttnRes, SwiGLU, bf16).

FP8

FP8 training was not tried in the paid run: the owner decided (2026-09-29) not to spend A/B time on the FP8 pair, so the run is bf16. The FP8 option exists in the training code (quipu/fp8.py) and was only checked on the laptop.

Evaluation

Bits per byte per language

Bits per byte (lower is better; comparable across tokenizers) on held-out text per language and held-out code. Definition: summed next-token loss in bits over the UTF-8 bytes of the target tokens. Every <|endoftext|> document separator is a target too: its loss counts but it is 0 bytes of text, so these numbers are slightly higher (more pessimistic) than a separator-free measure.

Code: HumanEval and MBPP

not yet measured (published numbers of reference models, for when it is:)

Expert usage

Routing frequency per expert, per layer and language (16 layers x 64 experts): the full tables and a heatmap are in the repository (results/experts). Here, dead: < 10% of the balanced target over the evaluated tokens; overloaded: > 300%. These are the trainer's health-alert bounds (quipu.train MOE_HEALTH_LOW / MOE_HEALTH_HIGH), applied to one evaluation pass; the training log's own "dead" count is stricter (experts with zero assignments in a logging interval), and its health alert needs a sustained window of steps out of bounds.

Expert specialisation by programming language (Python vs JavaScript / JSX, spec 7) was not measured: the code validation split carries no per-document language label, so code is one split in the expert-usage tables.

Samples

Greedy (temperature 0) continuations of the fixed prompts, final checkpoint, unedited: good and bad alike.

The history of the printing press (step_008129, greedy)

The history of the printing press is a fascinating one. It was the first printing press in the world, and it was the first to print on paper. It was also the first to print on paper.
The first printing press was invented in 1600 by the French scientist Jean Baptiste de Vaux. It was a machine that was used to print on paper. It was used to print on paper, and

Photosynthesis is the process by which (step_008129, greedy)

Photosynthesis is the process by which plants convert sunlight into chemical energy.
The process of photosynthesis is a process that occurs in the plant’s cells. The process is controlled by the plant’s own cells, which are the cells that produce the food and oxygen that plants need.
The process of photosynthesis is a process that occurs in the plant’s cells. The process is controlled by the plant ...

In mathematics, a prime number (step_008129, greedy)

In mathematics, a prime number is a prime number that is divisible by 2.
- 10000000000000000000000000000000000000000000000000000000000000000

def fibonacci(n): (step_008129, greedy)

def fibonacci(n):
    """
    Return the fibonacci of n
    """
    if n == 1:
        return 1
    elif n == 2:
        return fibonacci(n-1)
    elif n == 3:
        return fibonacci(n-2)
    elif n == 4:
        return fibonacci(n-3)
    elif n == 

function debounce(fn, delay) { (step_008129, greedy)

function debounce(fn, delay) {
  var timeout;
  return function () {
    var context = this, args = arguments;
    var later = function () {
      timeout = null;
      fn.apply(context, args);
    };
    var callNow = immediate && !timeout;
    clearTimeout(timeout);
    timeout = setTimeout(later, delay);
    if (callNow) fn.apply(context, args);
  };
}

**

** (step_008129, greedy)

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>105年第十四任總統副總統及第九屆立法委員選舉</title>

<link rel="stylesheet" type="text/css" href="css/main.css">

</head>

<body>
<div id="main">
    <div id="main-body">
        <div

SELECT customer_id, COUNT(*) (step_008129, greedy)

SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*) FROM customer;
SELECT customer_id, COUNT(*)

Sejarah kota Jakarta dimulai (step_008129, greedy)

Sejarah kota Jakarta dimulai pada tahun 1960-an.
Pada tahun 1960-an, kota Jakarta menjadi kota yang paling terkenal di dunia. Kota ini menjadi kota yang paling terkenal di dunia.
Pada tahun 1960-an, kota Jakarta menjadi kota yang paling terkenal di dunia. Kota Jakarta menjadi kota yang paling terkenal di dunia.
Pada tahun 1960-an, kota

Fotosintesis adalah proses (step_008129, greedy)

Fotosintesis adalah proses yang sangat cepat dan cepat.
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
- Meningkatkan daya tahan tubuh
-

Malaysia terkenal dengan (step_008129, greedy)

Malaysia terkenal dengan keindahan alam semulajadi.
Kini, Malaysia menjadi negara yang paling terkenal dengan keindahan alam semulajadi.
Kini, Malaysia menjadi negara yang paling terkenal dengan keindahan alam semulajadi.
Kini, Malaysia menjadi negara yang paling terkenal dengan keindahan alam semulajadi.
Kini, Malaysia menjadi negara yang paling terkenal dengan

Cara terbaik untuk belajar bahasa baharu ialah (step_008129, greedy)

Cara terbaik untuk belajar bahasa baharu ialah dengan menggunakan bahasa yang dipilih.
Bagi pelajar yang ingin belajar bahasa baharu, anda boleh memilih bahasa yang dipilih.
Bagi pelajar yang ingin belajar bahasa baharu, anda boleh memilih bahasa yang dipilih.
Bagi pelajar yang ingin belajar bahasa baharu, anda boleh memilih bahasa yang dipilih.
Bagi pelajar yang ingin belajar bahasa baharu, anda  ...

光合作用是植物 (step_008129, greedy)

光合作用是植物性的，它的成分是由植物性的，它的成分是由植物性的，它的成分是由植物性的，它的成分是由植物性的，它的成分是由植物性的，它的成分是由植物性的，它的成分是由植物性的，

北京是中国的 (step_008129, greedy)

北京是中国的一个小镇，在一个无处不在的丧尸已经威胁到了人类的生活的世界，
在那些美妙的日子里，在一个无处不在的丧尸已经威胁到了人类的生活的世界，
在那些美妙的日子里，在一个无处不

台灣的夜市 (step_008129, greedy)

台灣的夜市，
這次的行程是由台北市政府推動的，
這次的行程是由台北市政府推動的，
這次的行程是由台北市政府推動的，
這次的行程是由台北市政府推動的，
這次的行程是由台北市政府推動的

學習程式設計的第一步是 (step_008129, greedy)

學習程式設計的第一步是讓你在使用的時候，在使用的時候，你可以在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在使用的時候，在

日本の四季は (step_008129, greedy)

日本の四季は、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り、日本の四季を見る限り

東京で一番有名な (step_008129, greedy)

東京で一番有名な「アラフォー」の名前を紹介します。
「アラフォー」は、アラフォーと呼ばれるアラフォーの名前を紹介します。
「アラフォー」は、アラフォーと呼ばれるアラフォーの名前を紹介します。
「アラフォ

한국의 전통 음식은 (step_008129, greedy)

한국의 전통 음식은 1990년대 초반부터 시작되었다.
1990년대 초반부터 시작된 음식은 1990년대 초반부터 시작되었다.
1990년대 초반부터 시작된 음식은 1990년대 초반부터 시작되었다.
19

서울은 대한민국의 (step_008129, greedy)

서울은 대한민국의 100% 세계적인 인구의 10%를 차지했다.
이에 따라 100% 세계적인 인구는 100%를 차지했다.
이에 따라 100% 세계적인 인구는 100%를 차지했다.
이에 따라 10

தமிழ் மொழி உலகின் (step_008129, greedy)

தமிழ் மொழி உலகின் மிகப்பெரிய வரலாற்று சிறப்புமிக்க தமிழர் வாழ்வை வாழ்வதற்கான ஒரு வழி.
இலங்கை தமிழர் வாழ்வை வாழ்வதற்கான ஒரு வழி.
இலங்கை தமிழர் வாழ்வை வாழ்வதற்கான ஒரு வழி.
இலங்கை தமிழர் வாழ்வை வாழ்வதற்கான ஒரு வழி.

சென்னை நகரம் (step_008129, greedy)

சென்னை நகரம்.
இதுகுறித்து தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல் அறிந்ததும், தகவல்

भारत की राजधानी (step_008129, greedy)

भारत की राजधानी दिल्ली में 1000 से अधिक लोगों को रोजगार मिलेगा।
दिल्ली में 1000 से अधिक लोगों को रोजगार मिलेगा।
दिल्ली में 1000 से अधिक लोगों को रोजगार मिलेगा।
दिल्ली में 1000 से अधिक लोगों को रोजग

प्रकाश संश्लेषण वह प्रक्रिया है (step_008129, greedy)

प्रकाश संश्लेषण वह प्रक्रिया है जो आपकी सेवा के लिए एक अच्छा विकल्प है।
- आपकी सेवा के लिए एक अच्छा विकल्प है।
- आपकी सेवा के लिए एक अच्छा विकल्प है।
- आपकी सेवा के लिए एक अच्छा विकल्प है।
- आपकी सेवा के लिए एक अच्छा विकल्प है।

Mera naam Rahul hai aur main (step_008129, greedy)

Mera naam Rahul hai aur main 20 saal ka hoon. Main 25 saal ka hoon aur meri umar 24 saal hai. Main dikhne me bahut hi sundar hoon aur meri height 5’6 inch hai. Main dikhne me bahut hi sundar hoon aur meri height 5’6 inch hai.
Meri height 5’6 inch hai aur main dikhne me bahut hi sundar hoon. Meri height

Aaj mausam bahut (step_008129, greedy)

Aaj mausam bahut hi jyada badhiya hai.
Aaj ke samay me har koi apne ghar me hi rahata hai. Aaiye jante hai ki ghar me rahane ke liye kya kya karna chahiye.
Kya aap ghar me rahate hai?
Aaj ke samay me ghar me rahane ke liye kya kya karna chahiye.
Aaj ke samay me ghar me rahane ke liye

Pakistan ka sab se bara shehar (step_008129, greedy)

Pakistan ka sab se bara shehar hai.
Bahut se log is shehar ko dekh kar khush hain.
Bahut se log is shehar ko dekh kar khush hain.
Bahut se log is shehar ko dekh kar khush hain.
Bahut se log is shehar ko dekh kar khush hain.
Bahut se log is shehar ko dekh kar khush hain.
Bah

Mujhe kitabein parhna (step_008129, greedy)

Mujhe kitabein parhna hai.
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki charcha
- Bachon ki

int4

Weight-only int4 (int4-groupwise-asym-v1, groups of 128 along the last dimension, fp16 scale + minimum per group) for the attention, shared and routed expert matrices; embedding bf16; router, norms and biases fp32. Dequantized to fp32 on load (the file is smaller; memory once loaded is not).

The first tokens of each evaluation split (count per row). bf16 = the weights rounded to bf16, arithmetic in fp32 (spec 12's baseline); fp32 = the trained weights as saved.

Standalone-loader parity (max |logit difference| against the training code): final (fp32) 1.5e-05; milestone step_008129 (bf16) 2.3e-05; int4 (dequantized) 1.2e-05.

Stepbuild score, int4 vs bf16: not yet measured (needs the chat/step fine-tune).

Usage

pip install torch safetensors tokenizers huggingface_hub

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("AneekC/quipu-moe-1B-A149M", "modeling_quipu_moe.py")
spec = importlib.util.spec_from_file_location("modeling_quipu_moe", path)
mq = importlib.util.module_from_spec(spec)
sys.modules["modeling_quipu_moe"] = mq
spec.loader.exec_module(mq)

model = mq.load("AneekC/quipu-moe-1B-A149M")                                    # fp32
# model = mq.load("AneekC/quipu-moe-1B-A149M", weights="model-int4.safetensors")  # int4 file
tok = mq.load_tokenizer("AneekC/quipu-moe-1B-A149M")
print(mq.generate(model, tok, "def fibonacci(n):", max_new_tokens=60))

This is not a transformers model; there is no AutoModel class for it. The loader runs the experts one after another without a KV cache: correct, not fast.

Limitations

• Small: 148,660,992 parameters are active per token. It states false things with confidence; do not use its output as information.
• The nine languages other than English together are 12.0% of the training tokens (English 28.0%, code 60.0%; shard manifest, mix achieved). Expect weaker output in them than in English, and some mixing of languages; the bits per byte above show how much weaker, per language.
• Code correctness: not yet measured (HumanEval / MBPP above); treat generated code as untested.
• Expert specialisation by programming language (Python vs JavaScript / JSX, spec 7) was not measured: the code validation split carries no per-document language label, so code is one split in the expert-usage tables.
• Trained on web text and public code, which carry their biases; no safety tuning.

Credits and licence

Weights and code: Apache-2.0. Language identification of the training text used AneekC/lid-specialists-9plus1 (MIT). Training data: FineWeb-Edu and FineWeb-2 (ODC-By 1.0, © Hugging Face) and the permissively licensed subset of codeparrot/github-code-clean (files keep their licences). HumanEval (MIT) and MBPP (CC-BY-4.0) were used for evaluation only and filtered (substring / 13-gram matching) from the training data.