A NInfer v3 artifact of Thinking Machines'
Inkling 2 Mini, a looped
mixture-of-experts model of 26.4B parameters with 3.4B active per token, from IsValorum's
APEX-I MiniPlus V2.1 GGUF
with its Q8_0 vision tower and Unsloth's GGUF of the release's DSpark drafter, for NInfer-all,
the master branch of iamwavecut/ninfer-all. Every
matrix keeps the GGUF's ggml blocks byte for byte, so nothing is requantized; the engine multiplies
the blocks in place. Stock NInfer builds refuse this file: the Inkling 2 family exists only in that
line, and its Q8_0 vision tower needs commit 7fcbbb13 or later (release 2026-11-21.3).
At 12.8 GiB it is 2.6 GiB smaller than the UD-IQ4_XS artifact, whose routed experts are mostly NVFP4. By the engine's memory plan every expert fits on a 16 GB card beside an 8K-token context; no 16 GB card has run it yet.
It runs the whole model: one to eight concurrent requests with prompt-prefix reuse, the model's reasoning in its own blocks, its JSON tool-call blocks returned as OpenAI tool calls, structured output and strict tool arguments, typed decisions and reranking, images through the release's vision tower, speculative decoding with the release's DSpark drafter (by default while one request runs) or by context lookup, and the routed experts on the GPU, in pinned host memory or read from this file.
| component | representation |
|---|---|
| 24 stored layers run as 48 visits: the prelude (0-5), the core (6-17) three times with the same weights, the coda (18-23) | each visit with its own keys, values and convolution histories |
| attention: 24 query heads over 8 KV heads of 128, no rotation, a learned table of 1,024 distances, sinks; layers 5, 11, 17 and 23 over every position, the others over 1,024 | Q4_K query, key, value and distance projections; Q4_K output projections (Q6_K in layers 5, 11, 17 and 23); Q8_0 gates |
| MoE: 128 routed experts, 8 per token, sigmoid scores plus a selection bias, SiTU over [gate; up] (width 1152), no shared expert | IQ3_XXS [gate; up] and IQ4_NL down banks in the core layers, IQ3_S and Q5_0 in the prelude and the coda |
| token table; untied head (201,024 rows, 200,058 scored, soft cap 30) | Q4_K; Q6_K |
| router, convolutions, distance tables, depth-mixing queries, loop controllers, norms, expert scales, sink and dynamic-tap projections | FP32 (the GGUF's F32, BF16 and Q8_0 values, each held exactly) |
vision tower (vision): ViT with a class row, resized position rows, 2D-rotated heads, 2 x 2 merging, two-layer projector, up to 4,096 positions an image |
Q8_0 matrices, FP32 vectors, BF16 patch projection, from the repository's mmproj-Inkling-2-Mini-Q8_0.gguf |
DSpark drafter (dspark): three non-causal blocks over the target's tapped states, a recurrent Markov head of rank 256 |
Q8_0 from Unsloth's dspark-Inkling-2-Mini-Q8_0.gguf |
| tokenizer, chat template, generation config | thinkingmachines/Inkling-2-Mini at revision ed40e7da |
One .ninfer file of 13,740,465,920 bytes (12.8 GiB; SHA-256 42dd3b8731e9a7015d231803b56d8271ee610c4a61a3c78ef5a2ddd37f013b07), next to its conversion
report, SHA256SUMS and NOTICE: 1,173 objects, among them 817 FP32, 179 Q8_0, 117 Q4_K, 12
IQ3_S, 12 IQ3_XXS, 12 Q5_0, 12 IQ4_NL, 5 Q6_K and 2 BF16 tensors.
Conversion command, from the repository's tree at 7fcbbb13, with IsValorum's repository at
revision 2d412f7b (Inkling-2-Mini-APEX-I-MiniPlus-V2.1.gguf, sha256 1b86d2e8…;
mmproj-Inkling-2-Mini-Q8_0.gguf, fd84b797…), Unsloth's at ae9327c5
(dspark-Inkling-2-Mini-Q8_0.gguf, 2ec78510…; IsValorum's repository has no drafter) and the
Hugging Face text files at ed40e7da:
mkdir -p hf
for f in config.json tokenizer.json tokenizer_config.json chat_template.jinja generation_config.json; do
curl -fsSL -o hf/$f https://huggingface.co/thinkingmachines/Inkling-2-Mini/resolve/main/$f
done
hf download IsValorum/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-GGUF \
Inkling-2-Mini-APEX-I-MiniPlus-V2.1.gguf mmproj-Inkling-2-Mini-Q8_0.gguf --local-dir .
hf download unsloth/Inkling-2-Mini-GGUF dspark-Inkling-2-Mini-Q8_0.gguf --local-dir .
python3 -m tools.convert --model hf --recipe inkling2_gguf \
--source gguf=Inkling-2-Mini-APEX-I-MiniPlus-V2.1.gguf \
--source vision=mmproj-Inkling-2-Mini-Q8_0.gguf \
--source dspark=dspark-Inkling-2-Mini-Q8_0.gguf --components text,vision,dspark \
--name Inkling-2-Mini-APEX-I-MiniPlus-V2.1 \
--out Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer --device cpu
The conversion took 112 seconds on a rented A40's container.
NInfer-all runs on Linux and Windows with NVIDIA GPUs from Volta to Blackwell. Take a
release build or the Docker image
(cuda13.4, or cuda12.9 for a Tesla V100) from release 2026-11-21.3 on; earlier builds refuse its Q8_0
vision tower. Download the file once:
hf download WaveCut/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-NInfer-v3 \
Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer --local-dir models
Each server below speaks the OpenAI and Anthropic protocols on port 8080. The repository's launch
configs (configs/inkling2-mini.ini and configs/inkling2-mini-host.ini, in /opt/ninfer/configs
inside the image) take this file from the command line, which wins over the config's own model
path. DSpark drafts while one request runs; --spec none turns it off and does not load the
drafter.
Every expert on the GPU (24 GB and more: images, DSpark, a 64K-token context):
ninfer-serve models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config configs/inkling2-mini.ini
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -w / \
-v "$PWD/models:/models" -v ninfer-cache:/cache ghcr.io/iamwavecut/ninfer-all:cuda13.4 \
serve /models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config /opt/ninfer/configs/inkling2-mini.ini
On a 16 GB card add --max-context 8192: with the tower and the drafter one sequence plans
14.7 GiB of the GPU, 13.8 GiB with --spec none and without --vision; at 32K tokens 15.5 and
14.5 GiB. No 16 GB card has run this file yet.
Experts in system RAM (8 to 16 GB cards: 10.9 GiB of pinned host memory for the routed experts, a cache of the most used ones on the GPU, a 32K-token context):
ninfer-serve models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config configs/inkling2-mini-host.ini
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -w / \
-v "$PWD/models:/models" -v ninfer-cache:/cache ghcr.io/iamwavecut/ninfer-all:cuda13.4 \
serve /models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config /opt/ninfer/configs/inkling2-mini-host.ini
The GPU keeps 1.16 GiB of weights, two 1,002 MiB layer banks and the cache; CPU threads compute a
share of the cache misses. On an 8 GB card add --max-context 12288 (the UD-IQ4_XS artifact, with
larger banks, ran so on an RTX 5060; this one has not run on a card that small).
Experts read from the file (little free RAM; the OS page cache keeps what fits):
ninfer-serve models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config configs/inkling2-mini-host.ini --expert-residency disk
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -w / \
-v "$PWD/models:/models" -v ninfer-cache:/cache ghcr.io/iamwavecut/ninfer-all:cuda13.4 \
serve /models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--config /opt/ninfer/configs/inkling2-mini-host.ini --expert-residency disk
One prompt from the CLI:
ninfer models/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-ninfer-v3.ninfer \
--prompt "Write a binary search in Rust."
reasoning_content. reasoning_effort sets the template's thinking
level (none to xhigh). Thinking budgets do not apply: the model has no early-close control.{"name":...,"args":{...}}) comes back as an OpenAI tool
call; with --structured-output, strict tools and a response format beside tools hold under
their grammars, and a forced tool_choice opens the call block for the model.Not available for this model: quantized --kv-dtype formats (the cache is F16), MTP, DFlash and
n-gram copy drafts, rotation scaling (the attention has none), thinking budgets, video, pipeline
stages. Several GPUs can divide the routed experts (--tensor-parallel).
No task benchmarks were run on this artifact. On a rented A40, November 21, 2026:
ninfer-perplexity --kv-dtype bf16, 9,718
scored tokens): a mean NLL of 1.870942 (perplexity 6.494411), against 1.859871 (6.422907) for
the UD-IQ4_XS artifact,
+0.60%.examples/cli/messages/image_natural.json, it answers 24 and the right.ninfer_bench, one sequence, F16 KV, three repetitions, on a rented A40 (300 W, PCIe 4.0 x16,
CUDA 12.8), every expert on the GPU, in tok/s:
| Prefill 2,048 | Prefill 10,000 | Decode 128 | Decode after 10,000 |
|---|---|---|---|
| 2,776 | 2,128 | 104.3 | 90.2 |
The CLI's 384-token greedy essay: 103.1 tok/s with --spec none, 115.6 with DSpark (1.69 tokens a
round). The UD-IQ4_XS artifact gave 2,432, 1,897, 85.3 and 75.5 on another A40 host, and 84.4 and
97.3 on the essay. The A40's cache held every expert in the host and disk runs, so their speeds
(92.4 and 93.0 tok/s on the essay) do not show a small card's.
Model, tokenizer, chat template, generation config, vision tower and DSpark drafter: Thinking
Machines, Inkling 2 Mini, under the
Apache License 2.0. The APEX-I MiniPlus V2.1
quantization and its Q8_0 vision tower: IsValorum, under the Apache License 2.0. The GGUF
conversion of the drafter: Unsloth, under the Apache License 2.0. Attribution notices are
collected in NOTICE. This repository only re-packs those weights into NInfer's
container.
IsValorum/Inkling-2-Mini-APEX-I-MiniPlus-V2.1-GGUF