Files are up (updated 21 Nov, 18:10 AEDT). The quant passed its held-out quality and refusal gates and a quick task screen against its parents (below). Decode is now 93 tok/s with the latest Xolotl (was 59.7).
stage (21 Nov 2026, AEDT) status abliteration (ARA) + refusal gate done: 6/100 harmful, 1/100 harmless refused quantisation (~14 GB) done: 13.93 GB GGUF, verified (744 tensors, 0 problems) held-out KL / top-1 vs the parent done: KL 0.013, top-1 96.3 % vs the abliterated parent refusal gate on the quant done: 4/100 harmful, 1/100 harmless refused speed through the real server done: 59.7 tok/s decode, 1,162 tok/s prefill at 8K (RTX 4090); 93 tok/s decode with Xolotl 58ff5ddbb quick task screen vs parents done: MMLU-100 76 % (original 73 %), HumanEval, tool choice, agent loops on par Gauntlet v3 (agentic coding) paused (see Quality) upload done: model, DSpark drafter, vision projector (sha256-verified)
Thinking Machines' Inkling 2 Mini with the refusal direction edited out of the weights, then requantised with Indra's Mirror's mixed-precision XQ pipeline. A 26B-parameter looped MoE with 5.4B active parameters per token (about 3.1B distinct weights; layers 6-17 run 3 times), 128 routed experts at top-8, and a 1M-token context.
Runs on Xolotl, an engine built for this class of model and machine, on top of Strata. The forward graph is ported from llama.cpp (PR #32694 by Pedro Cuenca, MIT); the engine around it is Xolotl's own.
| file size | 13.93 GB |
| routed experts | mixed per layer, ~4.0 bits/weight on average; 13 of 24 layers keep the official NVFP4 experts bit for bit |
| dense backbone | 8-bit (Q8_0) |
| arch / tensors | inkling2_loop, 24 physical layers, 48 cache slots, 744 tensors |
Inkling 2 Mini is a 24-layer decoder-only transformer with sparse MoE feed-forward blocks. Its middle 12 layers (layers 6 to 17) run three times with shared weights, so each token gets 48 layer applications out of 24 layers of weights, and the cache carries 48 slots.
| total / active | 26B total, 5.4B active per token (3.1B distinct weights per token) |
| loop | layers 6 to 17 run 3 passes, 48 cache slots; passes 2 and 3 read the previous pass through per-layer low-rank loop controllers |
| MoE | 128 routed experts, top-8, no shared expert, sigmoid router with a selection bias, SiTU activation |
| attention | sliding window 1024 with a global layer every 6th (5, 11, 17, 23); no RoPE: a learned relative-position bias per head, with log-scaled positions on the global layers |
| extras | short causal convolutions on K, V, the attention output and the FFN output; block attention residuals that mix earlier depths; final logit softcap 30 |
| vocab | 200,058 |
| context | 1M tokens (the trained key says 1048576) |
| official precision | routed experts NVFP4 on layers 0 to 19, MXFP8 on 20 to 23, everything else BF16 |
The official checkpoint ships in that mixed precision. This build starts from it rather than from a dequantised copy, so the native FP4 experts can be carried through at their own precision.
Xolotl found and fixed a bug in the llama.cpp loop path along the way: passes 2 and 3 of the loop ran their experts without the per-expert NVFP4 scale when the GGUF keeps NVFP4 experts. The upstream converter dequantises, so upstream never meets that case. It is fixed in Xolotl's build.
Measured against the parent, which is the official checkpoint converted exactly (NVFP4 and MXFP8 kept) and run on the same engine, hardware and settings.
| parent | abliterated (before quant) | this (XQ) | |
|---|---|---|---|
| MMLU, 100-question sample (greedy, thinking off) | 73 % | 74 % | 76 % |
| MMLU, 30 questions, thinking on (high) | 93 % | - | 87 % |
| HumanEval, every 8th problem (21) | 21/21 | 20/21 | 19/21 |
| tool choice (10) / agent bug-fix loops (3) | 8/10, 3/3 | 8/10, 3/3 | 8/10, 3/3 |
| held-out top-1 agreement vs the original | n/a | 95.3 % | 94.7 % |
| held-out KL vs the original | n/a | 0.028 | 0.033 |
| held-out KL / top-1 vs the abliterated parent | n/a | n/a | 0.013 / 96.3 % |
| refusal gate, harmful prompts refused | not run | 6/100 | 4/100 |
| refusal gate, harmless prompts refused | not run | 1/100 | 1/100 |
Three columns on purpose: the abliteration and the quant are each checked for damage on their own.
With 100 and 30 questions, differences of a few points are noise: the quant sits with its parents on every task screen. Note for anyone measuring this model: its logits are chaotic at the rounding level (near-tied expert picks flip), so any engine change, even a different prefill chunk size, moves held-out KL by about 0.005 over 2K tokens; part of the 0.013 above is that floor.
Gauntlet v3 (Xolotl's multi-turn agentic coding set) is paused: on its long tasks the abliterated parent and the quant alike tend to deliberate for 10K+ thinking tokens on one step before acting, so a fair run needs a long thinking budget and many hours. It will be reported when it has been run that way.
Measured on an RTX 4090 through Xolotl's server (8K prompt, then 1,024 tokens): 1,162 tok/s prefill, 59.7 tok/s decode. The exact 22.9 GB parent does 1,131 / 57 on the same card, so the quant fits in 14 GB without losing speed.
Decode is far from the card's memory limit (a token reads about 3.4 GB, ~3.4 ms on a 4090): it is bound by the number of small GPU kernels per token (48 layer applications). First round of kernel fusion (Xolotl 58ff5ddbb): 67 -> 93 tok/s (256-token greedy; 93-96 tok/s in served evals), logits bit-identical to the unfused graph. More fusion is in progress, aiming at 100+ tok/s.
Landed in Xolotl (21 Nov):
--dspark, experimental): up to 7 drafted
tokens per step, with exact rollback of the model's recurrent convolution state. Still being tuned for speed.--kv-type q8_0|k8v4): the long-context path for 512K on one card.Next: vision, flash attention with the relative-position bias, and a grouped MoE kernel for speculative verify.
| file | size | what |
|---|---|---|
Inkling-2-Mini-XQ-Uncensored.gguf |
13.93 GB | the model |
mmproj-Inkling-2-Mini-XQ-Uncensored-F16.gguf |
0.64 GB | the official vision encoder, unchanged, for images (optional; engine support in progress) |
Inkling-2-Mini-XQ-Uncensored-DSpark-BF16.gguf |
0.64 GB | optional: the model's own DSpark (MTP) drafter for speculative decoding (--dspark), converted by unsloth |
Xolotl's xolotl-inkling engine runs any Inkling-2-Mini GGUF: the upstream converter's, llama.cpp's quants, or
this file.
git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
# build per docs/INSTALL.md, then serve with the Inkling config
The model is Thinking Machines Lab's, released with open weights under Apache 2.0. The Inkling 2 forward graph and the GGUF converter come from llama.cpp PR #32694 by Pedro Cuenca (MIT), which extends the earlier Inkling 2 work in llama.cpp and builds on ggml and llama.cpp by Georgi Gerganov and the llama.cpp contributors. Xolotl itself is built on Strata by Niko1221.
Apache 2.0, the same as the official model. The official card says, in its General Information section: "License: Apache 2.0", with the link to LICENSE.
thinkingmachines/Inkling-2-Mini