GGUF files of thinkingmachines/Inkling-2-Mini by AtomicChat: a 26.4B mixture-of-experts model (3.4B active, 128 experts with 8 active) with a looped core, where the middle 12 of its 24 layers run three times. It takes text and image input. License: Apache-2.0, the same as the base model (LICENSE in this repo).
AD (Atomic Dynamic) files give each tensor its own type. The experts hold 94% of the weights and carry the compression; attention and the loop controllers stay Q8_0. Quality is measured against our BF16: top-1 is how often the file picks the same next token as the original, and mean KLD is the divergence of the whole next-token distribution (lower is better).
| file | GB on disk | top-1 | mean KLD | notes |
|---|---|---|---|---|
BF16/ (2 parts) |
52.2 | — | — | the reference |
Inkling-2-Mini-Q8_0.gguf |
27.74 | 97.7% | 0.0033 | |
Inkling-2-Mini-AD-Q6_K.gguf |
23.79 | 97.6% | 0.0037 | Q8_0 quality, 4 GB smaller |
Inkling-2-Mini-Q6_K.gguf |
23.40 | 97.3% | 0.0047 | llama.cpp preset, no imatrix |
Inkling-2-Mini-Q5_K_M.gguf |
19.81 | 96.3% | 0.0079 | llama.cpp preset, no imatrix; still in the PR #32695 layout, does not load on b12600 yet |
Inkling-2-Mini-AD-Q5_K.gguf |
19.07 | 97.1% | 0.0052 | |
Inkling-2-Mini-Q4_K_M.gguf |
17.38 | 94.7% | 0.0165 | llama.cpp preset, no imatrix; still in the PR #32695 layout, does not load on b12600 yet |
Inkling-2-Mini-AD-Q4_K.gguf |
16.53 | 96.3% | 0.0080 | Q5_K_M quality, 3.3 GB smaller |
Inkling-2-Mini-AD-IQ4_XS.gguf |
15.00 | 95.8% | 0.0101 | |
Inkling-2-Mini-AD-IQ3_S.gguf |
13.34 | 94.2% | 0.0205 | |
Inkling-2-Mini-AD-IQ3_XXS.gguf |
12.58 | 93.3% | 0.0271 | |
Inkling-2-Mini-AD-IQ2_S.gguf |
11.56 | 91.3% | 0.047 | |
Inkling-2-Mini-AD-IQ2_XXS.gguf |
10.43 | 88.5% | 0.084 | |
Inkling-2-Mini-AD-IQ2_M.gguf |
8.67 | 85.7% | 0.130 | expert down Q2_0 except the 8 busiest layers |
Inkling-2-Mini-AD-IQ2_XXS-Q2.gguf |
7.90 | 84.3% | 0.158 | |
Inkling-2-Mini-AD-IQ1_M.gguf |
7.26 | 81.1% | 0.236 | 1-bit |
Inkling-2-Mini-AD-IQ1_S.gguf |
6.88 | 79.5% | 0.278 | 1-bit |
mmproj-Inkling-2-Mini-F16.gguf / -BF16.gguf |
0.64 | vision projector (image input) | ||
mmproj-Inkling-2-Mini-Q8_0.gguf |
0.35 | smaller vision projector, checked with AD-IQ1_S on b12603 |
Even Q8_0 matches the BF16's top token only 97.7% of the time on this model, so 97–98% is the ceiling here, not 99%+. Our guess is that rounding adds up over the 48 layer passes of the loop; we have not tested it.
llama-perplexity --kl-divergence, built from PR #32695. Every file was measured against the same BF16 logits, made once on an H100; the files themselves ran on H100 and A10 GPUs with that one build. The table, the importance matrix and the per-tensor recipes are in AtomicChat/Inkling-2-Mini-GGUF-metrics.patches/q2_0-imatrix.patch; llama.cpp's own Q2_0 quantizer ignores the matrix).The GGUF files in this repository have been updated to use the merged Inkling 2 loop architecture from llama.cpp PR #32694.
Requirements: llama.cpp b12600 or newer.
The previous implementation from PR #32695 is no longer required.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-server \
-m Inkling-2-Mini-AD-Q4_K.gguf \
--jinja \
-c 32768 \
-ngl 99
Important: Do not mix GGUF files from the old PR #32695 layout with the merged PR #32694 implementation.
reasoning_effort: none, minimal, low, medium, high, xhigh (default 0.9). Pass it as "chat_template_kwargs": {"reasoning_effort": "low"} on /v1/chat/completions. The vendor notes that none does not switch reasoning off completely.none and high, a tool call, an image question, and a reply in Chinese.thinkingmachines/Inkling-2-Mini