Inkling 2 Mini is Thinking Machines Lab's 26.4B-parameter looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for Apple silicon in mlx-vlm format: 14.8 GiB of weights, for Macs with 32 GB of memory (24 GB is tight).
The original 4-bit experts untouched, the four 8-bit expert layers requantized to 4-bit with error compensation, everything else 8-bit. No measurable loss against the original, 3.4 GiB smaller and about 20% faster.
The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the original if you have 32 GB or more and want the reference weights. These builds are for less memory.
| Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval |
|---|---|---|---|---|---|
| original, loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 |
| 4bit (this repo) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 |
| 3bit | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 |
| 2bit | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 |
| Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt |
|---|---|---|---|---|---|
| original, loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB |
| 4bit (this repo) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB |
| 3bit | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB |
| 2bit | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB |
All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task sets are small: they separate working builds from broken ones, and differences of a few problems between rows are noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in.
mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source:
pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm"
mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-4bit \
--prompt "Write a Python function that merges overlapping intervals." \
--reasoning-effort low --max-tokens 1024
Add --image photo.jpg for image input. In Python:
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("VisualInference/Inkling-2-Mini-MLX-4bit")
prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low")
raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text
message = processor.tokenizer.parse_response(raw, prefix="") # reasoning_content, content, tool_calls
print(message["content"])
The repository includes the model's bundled DSpark drafter (mtp.safetensors, unchanged). Pass
--draft-model VisualInference/Inkling-2-Mini-MLX-4bit to try speculative decoding; the speed section shows it is not a dependable gain
on an M4 Pro.
reasoning_effort (none, minimal, low, medium, high,
xhigh) and defaults to high when the client sends nothing. For agent loops low or medium is a better
default: each step is several times faster. none does not fully disable reasoning.max_tokens. With a small limit the model can spend it all on
reasoning and return an empty answer. Allow at least 1,000 tokens per turn.generation_config.json sets temperature 1.0. The numbers on this card use
temperature 0 so that builds can be compared.<|content_model_end_sampling|> (id 200006). <|end_message|> closes each
message inside a turn and is not a stop token; do not add it as a stop string.The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible server on loopback:
mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-4bit
Requests to /v1/chat/completions with tools return tool_calls (arguments as a JSON string),
finish_reason: "tool_calls", and the reasoning in reasoning_content.
This needs mlx-vlm at commit 434a1bed (2026-11-21, pull request 2783) or later, which added the tool-call parser
for Inkling 2; the install line above gets it. Older checkouts return neither tool_calls nor content. The patch
this card shipped before that commit is no longer needed and has been removed.
Without a server you can parse tool calls yourself: generate with skip_special_tokens=False and call
processor.tokenizer.parse_response(text, prefix=""), which uses the response template shipped in
tokenizer_config.json.
What was tested, with this repository downloaded from the Hub and stock mlx-vlm at commit 0d16abeb:
content: the malformed call described under Limitations.terminal, read_file, write_file, patch, search_files) in a real repository: explain the
project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a
script. A task passes only if the result checks out on disk.For Hermes, point a custom provider at the server (transport: chat_completions, the model id the server reports)
and set extra_body: {reasoning_effort: medium} on the provider: Hermes sends no reasoning effort to custom
endpoints, so the template's default, high, applies otherwise (the Hermes runs above used that default). Hermes
refuses endpoints that declare a context window under 64K tokens.
GSM8K and HumanEval in the first table were measured a day earlier on mlx-vlm commit 111ac668; those requests carry
no tools. The tool, JSON and Hermes rows were first measured there too, with a patch for the then missing parser, and
were repeated on stock mlx-vlm for this card.
| Part of the model | Original | This build |
|---|---|---|
| Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | unchanged: the original NVFP4 codes and block scales, bit for bit |
| Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 4-bit affine, groups of 64 (4.5 bits per weight) |
| Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 |
| Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged |
DSpark drafter (mtp.safetensors) |
bf16 | unchanged |
Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that saw few calibration tokens are regularised towards plain rounding.
On the calibration statistics, the squared output error of the gate and up projections, relative to the squared output, averages 0.0035 at 4-bit against 0.0070 for round-to-nearest.
config.json lists the quantization of every module explicitly, and extras/ has the recipe, the scripts, the
build manifest with file hashes, and the raw results behind this card.
The runtime matches the reference. Before quantizing anything, mlx-vlm's implementation was compared with the Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores 2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations. Expert dequantization, image preprocessing and chat-template token ids match the reference exactly.
16-bit activations reshuffle expert routing, harmlessly. The reference's own bf16 run differs from its float32 run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount. Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with float32). For this model, logit agreement with a reference says little once any perturbation is present, so the builds are judged on held-out loss and on tasks.
Activation quantization is not needed. The original's metadata lists NVFP4/MXFP8 quantization of the expert inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by -0.0011 to +0.0027 nats across three variants of the experiment: it does not help.
Held-out loss. 40,806 assistant-side tokens the calibration never saw (model-generated conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired bootstrap intervals over sequences. This build: +0.0017 ± 0.0013 nats per token over the original (1.4497 against 1.4480).
Tasks. GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval (40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0. This build: GSM8K 45 correct, 2 wrong, 3 cut off; HumanEval 36 correct, 0 wrong, 4 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code.
Cached decoding. Generation runs through key/value and convolution caches one token at a time, which rounds differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL divergence of 0.05 nats and agree on the top token 97% of the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no worse (0.791 against 0.800 nats per token). The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose under another.
The saved files are the tested model. After writing, the checkpoint was reloaded and its logits compared with the in-memory model: identical.
M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy:
| Prompt | Drafter | Prompt speed | Decode speed |
|---|---|---|---|
| short question (28 tokens) | off | 105 tok/s | 31.1 tok/s |
| short question (28 tokens) | DSpark | 118 tok/s | 18.7 tok/s |
| summarise a document (1,024 tokens) | off | 428 tok/s | 27.6 tok/s |
| summarise a document (1,024 tokens) | DSpark | 434 tok/s | 18.8 tok/s |
| summarise a long document (4,023 tokens) | off | 382 tok/s | 26.5 tok/s |
| summarise a long document (4,023 tokens) | DSpark | 373 tok/s | 21.8 tok/s |
Peak memory without the drafter, by prompt length and prefill step (prefill_step_size in Python,
--prefill-step-size for the server):
| Prompt | Prefill step | Prompt speed | Peak memory |
|---|---|---|---|
| 224 tokens | 1024 | 333 tok/s | 15.5 GiB |
| 4,023 tokens | 1024 | 401 tok/s | 19.1 GiB |
| 4,023 tokens | 256 | 347 tok/s | 16.2 GiB |
| 8,024 tokens | 256 | 331 tok/s | 16.5 GiB |
| 16,024 tokens | 256 | 325 tok/s | 16.9 GiB |
A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a small loss in prompt speed. Use it on a machine near its limit.
Drafter. With DSpark on, decoding was faster in 0 and slower in 3 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it.
Image check. One photograph and one chart, one question each. Both answers were correct, as with the original: the Pallas's cat on snow, and the chart's largest quarter, its value and the order reference.
0d16abeb it also loses the rest of the turn: a response that is not streamed carries the raw turn, control tokens included, as content, and a streamed response ends with an error event instead of a finish reason. The same happens when max_tokens runs out inside a tool call, such as a long write_file, so allow enough output tokens for the longest call. An agent harness
should treat either as a failed turn and retry. A fix is proposed in
mlx-vlm pull request 2786.skip_special_tokens=True the reasoning and the answer are concatenated in the returned text; use the
server, or parse the raw text with the response template, to separate them.Model and weights: Thinking Machines Lab, Apache 2.0, see the
original card and its
acceptable use policy. MLX runtime for this
architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream
revision ed40e7da625f.
model-00001-of-00003.safetensors 5.4 GBmodel-00003-of-00003.safetensors 5.3 GBmodel-00002-of-00003.safetensors 5.2 GBthinkingmachines/Inkling-2-Mini