An MLX 2-bit build of Thinking Machines' Inkling (975B-total / 41B-active MoE), quantized from the BF16 checkpoint. Omni: keeps the text decoder plus the vision (HMLP) and audio (dMel) towers. Self-contained.
We put this repo bundles the loader
package (inkling_mlx/), so it runs with just mlx + mlx-lm + transformers.
Community build for large-model-on-Apple-Silicon experiments. ~315 GB in 2-bit -> doesn't fit a single 192 GB Mac resident, so run it via multi-Mac (distributed MLX) or SSD expert-offload (see below — one 192 GB Mac, or even a 64 GB Mac slowly).
# pip install mlx mlx-lm transformers ; download this repo (it includes inkling_mlx/)
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer
model, config = load("./Inkling-mlx-2bit")
tok = AutoTokenizer.from_pretrained("./Inkling-mlx-2bit", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
~315 GB doesn't fit a 192 GB Mac resident but an MoE only fires 6 of 256 experts per token, so you can keep the always-needed weights in RAM (attention, shared experts, embeddings, norms, router, vision/audio towers ≈ 35 GB) and page the routed experts from SSD on demand, letting the OS page cache hold the hot ones. This makes the 2-bit omni build runnable on one 192 GB Mac Studio (and even generates on a 64 GB Mac, slowly).
Tool: github.com/huckiyang/mlx-moe-offload
(MIT) — a drop-in over mlx-lm's SwitchGLU. It also answers the mlx-lm feature request
ml-explore/mlx-lm#1438.
pip install mlx mlx-lm scipy pillow
git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e .
# 1) one-time: repack the stacked experts into a per-expert SSD store
# (copies the bundled inkling_mlx/ loader in, so the offload dir is self-contained)
python -m mlx_moe_offload.repack --build /path/Inkling-mlx-2bit --out /path/Inkling-mlx-2bit-offload
OFF=/path/Inkling-mlx-2bit-offload
# 2) generate — text, image, or audio (omni)
python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is"
python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?"
python examples/inkling_omni.py --offload-dir $OFF --audio q.wav --prompt "Transcribe in English:"
The prompt is prefilled in small --prefill-chunk blocks so only a bounded set of experts is
live at once (a lazy full-prompt prefill would page ~all experts per layer and OOM). Decode
speed is gated by the routing hit-rate + SSD bandwidth — check store.stats(). At 2-bit,
text is solid ("…capital of Japan is Tokyo") while image/audio are recognizable but degraded
by quantization; a higher-bit build sharpens multimodal quality.
Run entirely on a single 64 GB Mac with the routed experts paged from SSD:
| Modality | Input | Output |
|---|---|---|
| Text | The capital of Japan is |
Tokyo, but it is not |
| Audio | inkling_mlx/test-inkling.wav — "I love sushi and hamburger very much." |
I love sushi and hamburger very much ✅ |
The audio example transcribes clean English speech verbatim — the dMel audio tower + text backbone recover the sentence exactly, even at 2-bit. (Harder / accented clips can drift toward phonetically-similar wordings at 2-bit; a higher-bit build tightens this.)
Throughput is SSD-bound: on this 64 GB Mac (cold cache, expert LRU off) decode runs roughly 0.03–0.4 tok/s, with most of the wall-clock spent in a one-time prefill as the selected experts stream from disk. A 192 GB Mac Studio keeps about half the experts resident in the page cache and should reach the ~6–15 tok/s range.
Weights converted from thinkingmachines/Inkling.
We made a revised bundled inkling_mlx/ loader is vendored from PipeNetwork/inkling-mlx (Apache-2.0,
Copyright 2026 PipeNetwork) and David.
LICENSE + THIRD_PARTY_NOTICES.md are included.
model-00065-of-00065.safetensors 7.3 GBmodel-00031-of-00058.safetensors 6.2 GBmodel-00046-of-00058.safetensors 6.2 GBmodel-00015-of-00058.safetensors 6.2 GBmodel-00047-of-00058.safetensors 6.2 GBmodel-00057-of-00058.safetensors 6.1 GBmodel-00019-of-00058.safetensors 6.1 GBmodel-00056-of-00058.safetensors 6.1 GBmodel-00002-of-00058.safetensors 6.1 GBmodel-00009-of-00058.safetensors 6.1 GBmodel-00052-of-00058.safetensors 6.1 GBmodel-00012-of-00058.safetensors 6.1 GBmodel-00036-of-00058.safetensors 6.1 GBmodel-00021-of-00058.safetensors 6.1 GBmodel-00054-of-00058.safetensors 6.1 GBmodel-00042-of-00058.safetensors 6.1 GBmodel-00016-of-00058.safetensors 6.1 GBmodel-00040-of-00058.safetensors 6.1 GBmodel-00033-of-00058.safetensors 6.0 GBmodel-00004-of-00058.safetensors 6.0 GBmodel-00035-of-00058.safetensors 6.0 GBmodel-00049-of-00058.safetensors 5.9 GBmodel-00045-of-00058.safetensors 5.9 GBmodel-00007-of-00058.safetensors 5.8 GBmodel-00024-of-00058.safetensors 5.7 GBmodel-00026-of-00058.safetensors 5.3 GBmodel-00017-of-00058.safetensors 5.3 GBmodel-00029-of-00058.safetensors 5.2 GBmodel-00025-of-00058.safetensors 5.1 GBmodel-00030-of-00058.safetensors 5.1 GBmodel-00014-of-00058.safetensors 5.1 GBmodel-00027-of-00058.safetensors 5.1 GBmodel-00003-of-00058.safetensors 5.1 GBmodel-00022-of-00058.safetensors 5.1 GBmodel-00020-of-00058.safetensors 5.1 GBmodel-00005-of-00058.safetensors 5.1 GBmodel-00039-of-00058.safetensors 5.1 GBmodel-00043-of-00058.safetensors 5.1 GBmodel-00001-of-00058.safetensors 5.1 GBmodel-00023-of-00058.safetensors 5.1 GBmodel-00051-of-00058.safetensors 5.1 GBmodel-00028-of-00058.safetensors 5.1 GBmodel-00011-of-00058.safetensors 5.1 GBmodel-00008-of-00058.safetensors 5.1 GBmodel-00055-of-00058.safetensors 5.1 GBmodel-00010-of-00058.safetensors 5.1 GBmodel-00053-of-00058.safetensors 5.1 GBmodel-00001-of-00065.safetensors 5.0 GBmodel-00041-of-00058.safetensors 5.0 GBmodel-00034-of-00058.safetensors 5.0 GBmodel-00018-of-00065.safetensors 5.0 GBmodel-00027-of-00065.safetensors 5.0 GBmodel-00019-of-00065.safetensors 5.0 GBmodel-00006-of-00065.safetensors 5.0 GBmodel-00040-of-00065.safetensors 5.0 GBmodel-00020-of-00065.safetensors 5.0 GBmodel-00052-of-00065.safetensors 5.0 GBmodel-00050-of-00065.safetensors 5.0 GBmodel-00058-of-00065.safetensors 5.0 GBmodel-00007-of-00065.safetensors 5.0 GBmodel-00026-of-00065.safetensors 5.0 GBmodel-00031-of-00065.safetensors 5.0 GBmodel-00037-of-00065.safetensors 5.0 GBmodel-00038-of-00065.safetensors 5.0 GBmodel-00039-of-00065.safetensors 5.0 GBmodel-00004-of-00065.safetensors 5.0 GBmodel-00010-of-00065.safetensors 5.0 GBmodel-00030-of-00065.safetensors 5.0 GBmodel-00032-of-00065.safetensors 5.0 GBmodel-00024-of-00065.safetensors 5.0 GBmodel-00005-of-00065.safetensors 5.0 GBmodel-00053-of-00065.safetensors 5.0 GBmodel-00046-of-00065.safetensors 5.0 GBmodel-00047-of-00065.safetensors 5.0 GBmodel-00059-of-00065.safetensors 5.0 GBmodel-00014-of-00065.safetensors 5.0 GBmodel-00025-of-00065.safetensors 5.0 GBmodel-00043-of-00065.safetensors 5.0 GBmodel-00033-of-00065.safetensors 5.0 GBmodel-00051-of-00065.safetensors 5.0 GBmodel-00056-of-00065.safetensors 5.0 GBmodel-00023-of-00065.safetensors 5.0 GBmodel-00057-of-00065.safetensors 5.0 GBmodel-00017-of-00065.safetensors 5.0 GBmodel-00064-of-00065.safetensors 5.0 GBmodel-00060-of-00065.safetensors 5.0 GBmodel-00013-of-00065.safetensors 5.0 GBmodel-00011-of-00065.safetensors 5.0 GBmodel-00012-of-00065.safetensors 5.0 GBmodel-00063-of-00065.safetensors 5.0 GBmodel-00034-of-00065.safetensors 5.0 GBmodel-00006-of-00058.safetensors 5.0 GBmodel-00013-of-00058.safetensors 5.0 GBmodel-00038-of-00058.safetensors 5.0 GBmodel-00050-of-00058.safetensors 5.0 GBmodel-00036-of-00065.safetensors 5.0 GBmodel-00008-of-00065.safetensors 5.0 GBmodel-00015-of-00065.safetensors 5.0 GBmodel-00049-of-00065.safetensors 5.0 GBmodel-00002-of-00065.safetensors 5.0 GBmodel-00044-of-00065.safetensors 5.0 GBmodel-00009-of-00065.safetensors 5.0 GBmodel-00035-of-00065.safetensors 5.0 GBmodel-00054-of-00065.safetensors 5.0 GBmodel-00061-of-00065.safetensors 5.0 GBmodel-00028-of-00065.safetensors 5.0 GBmodel-00041-of-00065.safetensors 5.0 GBmodel-00029-of-00065.safetensors 5.0 GBmodel-00042-of-00065.safetensors 5.0 GBmodel-00021-of-00065.safetensors 5.0 GBmodel-00062-of-00065.safetensors 5.0 GBmodel-00003-of-00065.safetensors 5.0 GBmodel-00022-of-00065.safetensors 5.0 GBmodel-00055-of-00065.safetensors 5.0 GBmodel-00048-of-00065.safetensors 5.0 GBmodel-00045-of-00065.safetensors 5.0 GBmodel-00016-of-00065.safetensors 5.0 GBmodel-00018-of-00058.safetensors 5.0 GBmodel-00044-of-00058.safetensors 5.0 GBmodel-00037-of-00058.safetensors 5.0 GBmodel-00048-of-00058.safetensors 5.0 GBmodel-00032-of-00058.safetensors 5.0 GBmodel-00058-of-00058.safetensors 2.1 GB有文件?提交种子。
提交种子thinkingmachines/Inkling