Full-Spectrum Looped-MoE Systems Study, Custom Ampere (
sm_86) CUDA Kernels, Turbo-DSpark (mtp.safetensors) Speculative Decoding, Multi-Process Heretic Abliteration, Long-Context Scaling & Multi-Agent Concurrency on a Single 24 GB RTX 3090 (280 W).
thinkingmachines/Inkling-2-Mini (33.3B total parameters, ~2.13B active per physical pass, 48 depth visits via P=3 Looped Core)BF16 + NVFP4 + MXFP8 + mtp.safetensors): JoaoZaokk/Inkling-2-Mini-HereticC5_MidCoT_TriBoundary.hbin — KLD = 0.05913, WikiText-2 PPL = 8.9361, GSM8K CoT = 96.88%, Algorithmic Coding CoT = 100.00%, XSTest Safe False-Refusal = 7.5%)| File | Format / Role | Size | Recommended Use Case |
|---|---|---|---|
Inkling-2-Mini-Heretic-UD-Q4_K_XL.gguf |
Unsloth Dynamic 4-bit XL (NVFP4 core + Q8_0/Q6_K sensitive) |
16.42 GiB |
Highest-precision 4-bit (PPL = 8.9350) |
Inkling-2-Mini-Heretic-UD-Q4_K_M.gguf |
Unsloth Dynamic 4-bit Medium | 15.47 GiB |
Sweet-spot daily driver, 262K pure-GPU context & 24 concurrent agents (225.3 tok/s) |
Inkling-2-Mini-Heretic-UD-IQ4_XS.gguf |
Importance-Matrix 4-bit XS | 14.65 GiB |
Lowest-PPL 4-bit (PPL = 8.9293, 64.1 tok/s solo) |
Inkling-2-Mini-Heretic-UD-Q3_K_XL.gguf |
Unsloth Dynamic 3-bit XL | 12.70 GiB |
Best VRAM/Quality Pareto (PPL = 8.9406 — virtually lossless vs Q4_K_XL!), 262K+ pure-GPU context |
Inkling-2-Mini-Heretic-UD-Q2_K_XL.gguf |
Unsloth Dynamic 2-bit XL | 8.51 GiB |
524,288 (512K) pure-GPU context (17.8 GiB), 84.3 tok/s Turbo-DSpark, & 48 concurrent agents (247.5 tok/s peak) |
dspark-Inkling-2-Mini-Q8_0.gguf |
Native MTP Draft Model (mtp.safetensors, Q8_0) |
369 MB |
Turbo-DSpark speculative decoding (--spec-draft-n-max 3) |
dspark-Inkling-2-Mini-BF16.gguf |
Native MTP Draft Model (BF16) |
636 MB |
Lossless reference DSpark draft model |
mmproj-Inkling-2-Mini-Q8_0.gguf |
Vision Multimodal Projector (Inkling2V) |
338 MB |
Image + text multimodal inference |
patches/llama_inkling2_sm86.patch |
Custom CUDA sm_86 + Banded Q4_0/Q8_0 KV + Turbo-DSpark patch |
69 KB |
Complete llama.cpp source patch used in all benchmarks |
recipes/C5_MidCoT_TriBoundary.hbin |
Winning Dual-Side Heretic Subspace Recipe | 1.33 MB |
Exact orthonormal basis vectors applied to all checkpoints |
Inkling-2-MiniInkling-2-Mini (Inkling2LoopForConditionalGeneration) is a Looped Hybrid Attention-ShortConv Mixture-of-Experts multimodal model with a 1,048,576-token architectural context window:
24 physical layers $\rightarrow$ 48 execution visits at P=3):layers 0..5): Executed once (6 visits).layers 6..17): 12 physical layers executed P = 3 passes (36 visits). Passes p > 0 reset to the prelude state and inject learned depth-attention messages (loop_res_query, loop_kv_res_query) gated by rank-32 controllers:
$$x' = \big(1 - \tanh(F h)\big) \odot x + \tanh(G h) \odot m, \qquad h = D \big[\mathrm{RMS}(x);; \mathrm{RMS}(m)\big]$$layers 18..23): Executed once (6 visits; MXFP8 in layers 20..23 vs NVFP4 in layers 0..19).attn_res_block_size = 6): Every sublayer mixes historical 6-layer block outputs via softmax depth-attention (mix()).8 Global KV Slots across 48 Visits):20 physical layers (40 visits) use 512-token Sliding-Window Attention (SWA) with 16-tap relative bias.4 physical layers (5, 11, 17, 23) use global attention with log-position Q/R scaling $\tau(p) = 1 + \alpha \ln(\max(1, (p+1)/4096))$, learned attention sinks, and L2 Q/K/Rel clipping. Because layers 11 and 17 execute 3 passes inside the looped core, each sequence maintains 8 independent global KV cache slots (visits 5, 11, 17, 23, 29, 35, 41, 47).l_cache = 4): Applied on K, V, post-attention (attn_post_conv), and post-MoE (ffn_post_conv), with dynamic input-conditioned taps on global layers.top-10 sigmoid routing + SiTU activation).DSpark (mtp.safetensors): Yes — Inkling-2-Mini ships with a native 636 MB DSpark block-speculative draft network (5 SWA 512 layers fusing features from target visits [2, 14, 25, 36, 47] + recurrent markov_rank = 256 head drafting up to block_size = 8 tokens in parallel).sm_86) CUDA Kernels (patches/llama_inkling2_sm86.patch)All benchmarks were executed on 1× NVIDIA GeForce RTX 3090 (24 GB, capped at 280 W) using interleaved symmetric ABBA runs (stock vs sm86):
Q8_0 & Q4_0 KV Cache (fattn-banded.cu):fattn-banded.cu only accepted F16/BF16/F32 KV caches, forcing long-context quantized KV caches onto slow dense paths or multi-GB FP16 scratch buffers.Q8_0 & Q4_0 dequantization directly inside fattn_banded_mma_load_tile and coalesced fattn_banded_load_kv split-K decode (0 bytes global scratch VRAM, +57.42% faster Q4_0 KV prefill!).UE4M3 (NVFP4 Block Scale) Decode (common.cuh, vecdotq.cuh, mmvq.cu):ldexpf() calls on sm_86 with exact branchless IEEE-754 bit manipulation + streaming ld.global.cs.u32 NVFP4 loads, aligned 4-byte __ldg Q8_1 loads, and Ampere-tuned mmvq crossover thresholds.inkling_l2_clip_f32_kernel & inkling_loop_gate_f32_kernel (ggml-cuda.cu):clip() subgraph (192 calls/token) and 7-node control() epilogue (144 calls/token) into single zero-spill warp-shuffle CUDA kernels (REG:18 & REG:35, STACK:0).inkling2.cpp):48 pass-level depth-attention messages once per pass (pass_m_attn, pass_m_kv), cached cached_blocks_rms across each 6-layer Attention-Residual block, and eliminated redundant transpose launches in convolve during single-token decode.Benchmark (UD-Q4_K_M, 1× RTX 3090 @ 280 W) |
Stock llama.cpp (tok/s) |
sm_86 Patched (tok/s) |
Speedup | WikiText-2 PPL Parity |
|---|---|---|---|---|
tg64 (Solo Decode, F16 KV) |
51.93 |
63.53 |
+22.34% |
7.8663 vs 7.8516 |
pp8 (Micro-Batch / Spec Verify) |
155.72 |
166.23 |
+6.75% |
exact |
pp64 (Short Prompt Prefill) |
681.66 |
737.50 |
+8.19% |
exact |
pp512 (Medium Prefill) |
1407.33 |
1755.09 |
+24.71% |
exact |
pp2048 (F16 KV Prefill) |
1570.59 |
1855.78 |
+18.16% |
exact |
pp2048 (Q4_0 KV Banded MMA vs Dense) |
1148.33 |
1807.72 |
+57.42% |
7.8657 vs 7.8331 |
P = 1..5) & Picard Damping MatrixUsing our runtime INKLING2_NUM_PASSES, INKLING2_MSG_SCALE, and INKLING2_EXTRAP_DAMP hooks, we evaluated every recurrent pass configuration from P = 1 (24 visits) to P = 5 (72 visits):
| Configuration | Core Passes (P) |
Total Visits | pp512 (tok/s) |
tg64 (tok/s) |
VRAM (MiB) |
WikiText-2 PPL $\downarrow$ | GSM8K Acc $\uparrow$ |
|---|---|---|---|---|---|---|---|
P=1 (24 visits, 0 recur) |
1 |
24 |
3419.56 |
127.12 |
15996 |
19.6616 |
0.0% |
P=2 (36 visits, 1 recur) |
2 |
36 |
2164.38 |
77.65 |
16340 |
14.4286 |
12.5% |
P=2 + BoostMsg (msg=1.15) |
2 |
36 |
2171.09 |
78.27 |
16340 |
14.6489 |
6.2% |
P=3 Default (48 visits) |
3 |
48 |
1578.86 |
57.59 |
16454 |
8.4393 |
87.5% |
P=3 Damped (msg=0.75) |
3 |
48 |
1594.83 |
57.90 |
16454 |
8.4601 |
87.5% |
P=3 Boosted (msg=1.20) |
3 |
48 |
1605.52 |
57.78 |
16454 |
8.4333 |
68.8% |
P=4 Raw Extrap (60 visits) |
4 |
60 |
1266.01 |
45.82 |
16572 |
12.7942 |
0.0% |
P=4 Picard-Damped (damp=0.45) |
4 |
60 |
1268.53 |
45.72 |
16572 |
13.2829 |
0.0% |
P=5 Raw Extrap (72 visits) |
5 |
72 |
1055.30 |
37.92 |
16686 |
12.0291 |
18.8% |
P=5 Picard-Damped (damp=0.40) |
5 |
72 |
1055.34 |
37.61 |
16686 |
13.4827 |
6.2% |
[!IMPORTANT] Discrete Attractor Finding:
Inkling-2-Miniis not a contractively trained Universal Transformer;P = 3(48visits) is a sharp discrete fixed point (PPL = 8.4393,GSM8K = 87.5%). Truncating toP = 1doubles speed (127.12 tok/s) for ultra-fast classification/routing, whereas unrolling toP = 4, 5diverges off the Coda manifold even under Picard damping (0.40–0.45). KeepP = 3(msg_scale = 0.75–1.00) for all reasoning workloads.
DSpark / mtp.safetensors) & Turbo-DSpark OptimizationDSpark Bottlenecked at draft = 7 — and How We Fixed ItProfiling src/models/dspark.cpp on Ampere (sm_86) uncovered two major bottlenecks in the recurrent Markov draft head (build_inkling2_head, executed up to 7 sequential times per block):
492 MB target.output GEMVs (3.44 GB HBM traffic per draft block!): Line 961 computed target.output @ (dspark_hidden_out @ latent + b) ([2304, 201088]) at every Markov step. By pre-fusing $W_{\text{markov}} = W_{\text{target_out}} W_{\text{hidden_out}}^\top \in \mathbb{R}^{256 \times 201088}$ (Q8_0 = 54.7 MB, 9.0× smaller!) and $b_{\text{markov}} = W_{\text{target_out}} b_{\text{hidden_out}}$ once at initialization, we cut Markov head HBM reads from 3.44 GB to 383 MB.ggml_pool_1d(k=201024) Tie-Break (~13.3 ms wasted per block!): Lines 1111–1114 scanned 201,024 floats on a single CUDA warp (pool_1d + repeat + sub + step + scale + mul + argmax) 7 times solely to break ties toward the lowest token ID. Updating argmax.cu warp reductions (col < argmax on val == maxval) collapsed all 7 ops into a single 1024-thread ggml_argmax (0.10 ms).Turbo-DSpark Setting (UD-Q4_K_M) |
Mean tok/s |
Mean Accept | Math CoT (tok/s / Accept) |
Python Code (tok/s / Accept) |
JSON Schema (tok/s / Accept) |
|---|---|---|---|---|---|
draft = 0 (block=1) |
61.86 |
0.0% |
62.48 (0.0%) |
61.58 (0.0%) |
61.53 (0.0%) |
draft = 3 (block=4) |
68.83 |
60.6% |
79.58 (73.3%) |
67.82 (62.9%) |
59.09 (45.8%) |
draft = 5 (block=6) |
66.39 |
45.9% |
79.34 (57.6%) |
69.71 (51.4%) |
50.12 (28.6%) |
draft = 7 (block=8) |
60.24 |
35.9% |
75.57 (49.0%) |
61.63 (37.6%) |
43.53 (21.2%) |
280 W) vs Apple M5 Ultra / M6 (200–300 tok/s)Why do M5 Ultra / M6 unified-memory chips reach 200–300 tok/s single-stream while a 280 W-capped RTX 3090 runs single-stream at 62–66.6 tok/s solo / 74–84.3 tok/s with Turbo-DSpark — and then surges to 225–247.5 tok/s aggregate across parallel agents?
6 MB L2 Cache (nsys Verified): Because GA102 has only a 6 MB L2 cache while the 12 looped core layers occupy ~8 GiB, a batch-1 decode must re-stream active attention + top-10 MoE weights from GDDR6X 3 separate times per token (48 visits $\approx 3.59\text{ GB/tok}$ in Q4_K_M), plus execute ~2,500 sequential micro-kernels/token (~5.5 ms GPU wave-tail dispatch floor across 48 recurrent ShortConv + depth-attention visits).16–48 Agents = 208–247.5 tok/s!): Apple Silicon Ultra chips leverage massive 96–192 MB System-Level Cache (SLC) and low-overhead unified dispatch at batch=1, whereas on RTX 3090 (sm_86), running 16–48 concurrent agent slots (llama-server -np 32 -kvu) amortizes every weight read and kernel launch across all active agents simultaneously, reaching 247.48 tok/s on a single 280 W RTX 3090!We engineered and evaluated 4 distinct Heretic abliteration processes (H1..H4) and 5 principled combinations (C1..C5), probing all 49 depth slots (0..47 layer visits + 48 = result_norm) across direct Thinking effort level: 0 chat, <|content_thinking|> onset, mid-CoT safety transition, and post-thinking <|end_message|><|message_model|><|content_text|> boundaries:
layers 6..17): Measuring refusal vectors across Pass 0 (visit l), Pass 1 (visit l+12), and Pass 2 (visit l+24) of the exact same shared weights revealed that the 3 loop passes operate in nearly orthogonal subspaces (cos(P0, P1) = +0.093, cos(P0, P2) = +0.071 on layer 11; cos(P0, P2) = -0.159 on layer 15!). Averaging [P0, P1, P2] into a Rank-1 vector (H1, H3, H4) cancels out 2/3 of the projection (100% refusal), whereas targeting the exit pass (Pass 2 + Coda Rank-2 SVD) drops chat refusal from 100% to 12.5%–20.0%!result_norm (cos = +0.244..+0.305): Inside layers 6..23, direct-chat refusal (e0) and thinking-onset refusal (e9_on) are nearly collinear (cos = +0.97..+0.998), but at result_norm (output.weight / lm_head) they rotate sharply (cos(e0, mid_cot) = +0.244, cos(e0, e9_on) = +0.283)! Spanning the 4-boundary orthonormal subspace orth([d0_rn, dm_rn, dp_rn, d9_rn]) on the LM head (C5) cut <|content_thinking|> refusal from 100.0% down to 58.33% while keeping 100.0% Algorithmic Coding and 96.88% GSM8K!ReasonShield & ShortConv Anisotropy: Extracting the protected reasoning subspace Q_think strictly within the same Thinking effort level: 0.9 template (cot_math - benign_chat_think) prevents template-token leakage and preserves 97.97% composite reasoning and 7.5% XSTest safe false-refusal (identical to unmodified Base!).UD-Q4_K_M)| Model / Process | KLD $\downarrow$ |
Top-1 $\uparrow$ |
WikiText-2 PPL $\downarrow$ |
Chat Refusal $\downarrow$ | Think Refusal $\downarrow$ | XSTest False Refusal $\downarrow$ |
GSM8K CoT $\uparrow$ |
Coding CoT $\uparrow$ |
Reasoning Composite $\uparrow$ |
|---|---|---|---|---|---|---|---|---|---|
| Base (Unmodified) | 0.00000 |
100.00% |
8.4262 |
100.0% |
100.0% |
7.5% |
100.00% |
100.00% |
100.00% |
| H1: WriteOnly-Parametric | 0.01420 |
93.16% |
8.6056 |
100.0% |
100.0% |
7.5% |
96.88% |
100.00% |
97.97% |
| H2: DualSide-LateSVD2 | 0.02651 |
90.33% |
8.9542 |
17.5% |
75.0% |
5.0% |
96.88% |
100.00% |
97.97% |
| H3: ReasonShield-Biorth | 0.01194 |
92.87% |
8.6126 |
100.0% |
100.0% |
10.0% |
100.00% |
100.00% |
100.00% |
| H4: ShortConv-LoopGuard | 0.03139 |
90.72% |
8.9343 |
100.0% |
100.0% |
10.0% |
100.00% |
100.00% |
100.00% |
| C1: H2 + H3 | 0.02090 |
91.11% |
8.7369 |
22.5% |
83.3% |
7.5% |
96.88% |
91.67% |
95.05% |
| C2: H2 + H4 | 0.03829 |
89.55% |
9.0823 |
22.5% |
83.3% |
7.5% |
96.88% |
91.67% |
95.05% |
| C3: H1 + H3 + H4 | 0.02890 |
90.33% |
8.7315 |
100.0% |
100.0% |
10.0% |
93.75% |
91.67% |
93.02% |
| C4: H2 + H3 + H4 (LoopShield) | 0.05067 |
87.70% |
8.8821 |
32.5% |
75.0% |
10.0% |
96.88% |
100.00% |
97.97% |
| C5: MidCoT-TriBoundary + ReasonShield (Winner) | 0.05913 |
88.38% |
8.9361 |
20.0% |
58.3% |
7.5% |
96.88% |
100.00% |
97.97% |
| Quantization | File Size | VRAM (8K + DSpark) |
pp512 (tok/s) |
tg64 Solo (tok/s) |
Turbo-DSpark Math (tok/s) |
Turbo-DSpark Code (tok/s) |
Turbo-DSpark JSON (tok/s) |
WikiText-2 PPL $\downarrow$ |
KLD vs Base Q4_K_M |
|---|---|---|---|---|---|---|---|---|---|
Heretic-UD-Q4_K_XL |
16.42 GiB |
18536 MiB |
1723.19 |
62.26 |
73.98 |
60.23 |
53.12 |
8.9350 |
0.05931 |
Heretic-UD-Q4_K_M |
15.47 GiB |
17564 MiB |
1743.92 |
62.13 |
74.11 |
68.49 |
55.62 |
8.9361 |
0.05913 |
Heretic-UD-IQ4_XS |
14.65 GiB |
16728 MiB |
1779.38 |
64.11 |
70.54 |
66.38 |
59.23 |
8.9293 |
0.05892 |
Heretic-UD-Q3_K_XL |
12.70 GiB |
14888 MiB |
1738.79 |
62.35 |
73.56 |
72.73 |
60.76 |
8.9406 |
0.05913 |
Heretic-UD-Q2_K_XL |
8.51 GiB |
10672 MiB |
1836.46 |
66.56 |
84.30 |
69.95 |
65.88 |
11.6550 |
0.38747 |
[!TIP]
UD-Q3_K_XL(12.70 GiB) is an extraordinary Pareto sweet spot: becauseInkling-2-Mini'slayers 0..19routed experts are natively trained inNVFP4,UD-Q3_K_XLachievesWikiText-2 PPL = 8.9406andKLD = 0.05913— virtually indistinguishable fromUD-Q4_K_XL(8.9350) while saving3.72 GiBVRAM! Meanwhile,UD-Q2_K_XL(8.51 GiB) unlocks524,288(512K) pure-GPU context and48+concurrent agents on a single 24 GB card.
128K $\rightarrow$ 524,288 (512K) Pure-GPU & 1,048,576 (1M) Context AnalysisBecause Inkling-2-Mini loops layers 11 and 17 three times (P=3), it allocates 8 global KV cache slots (~1,152 MiB per 128K tokens in q4_0/q4_0, plus self_rel_pos/self_attn_mask buffers — use -b 256 -ub 256 -kvu above 256K to avoid multi-GB mask allocations!):
-ngl 999): 262,144 (256K) on UD-Q4_K_M (21,512 MiB) and UD-Q3_K_XL (18,838 MiB), and 524,288 (512K) on UD-Q2_K_XL (17,876 MiB, leaving 6.3 GiB free — up to ~786K pure-GPU!).1,048,576 (1M) Context: Requires ~25.9 GiB total (8.5 GiB Q2_K_XL weights + ~17.3 GiB 8-visit q4_0 KV & relative position tables), which runs cleanly on 32 GB+ GPUs, Dual-3090 (--tensor-split 1,1), or on a single 24 GB RTX 3090 + host RAM by offloading 6 expert layers (-ot "blk.(1[8-9]|2[0-3]).ffn_.*_exps.=CPU").| Quantization | Context Window (-c) |
KV Cache (K/V) |
DSpark |
24 GB Pure-GPU Status | Idle VRAM (MiB) |
Peak VRAM (MiB) |
Long Prefill (tok/s) |
Decode (tok/s) |
Passkey Needle |
|---|---|---|---|---|---|---|---|---|---|
Heretic-UD-Q4_K_M |
131,072 |
q4_0/q4_0 |
No |
OK | 18668 |
18696 |
1201.13 |
13.3 |
PASS |
Heretic-UD-Q4_K_M |
262,144 |
q4_0/q4_0 |
No |
OK | 21484 |
21512 |
1202.69 |
13.51 |
PASS |
Heretic-UD-Q3_K_XL |
131,072 |
q4_0/q4_0 |
No |
OK | 15992 |
16022 |
1173.35 |
14.34 |
PASS |
Heretic-UD-Q3_K_XL |
262,144 |
q4_0/q4_0 |
No |
OK | 18808 |
18838 |
1168.43 |
14.17 |
PASS |
Heretic-UD-Q4_K_M |
524,288 |
q4_0/q4_0 |
No |
OOM | - |
- |
- |
- |
- |
Heretic-UD-Q4_K_M |
655,360 |
q4_0/q4_0 |
No |
OOM | - |
- |
- |
- |
- |
Heretic-UD-Q2_K_XL |
524,288 |
q4_0/q4_0 |
No |
OK | 17854 |
17876 |
1007.47 |
14.55 |
PASS |
Heretic-UD-Q2_K_XL |
1,048,576 |
q4_0/q4_0 |
No |
OOM | - |
- |
- |
- |
- |
Heretic-UD-Q2_K_XL |
1,048,576 |
q4_0/q4_0 |
Yes |
OOM | - |
- |
- |
- |
- |
--parallel 1..48, -kvu Unified KV Pool)| Quantization | Shared KV Pool (-c) |
Concurrent Agents (-np) |
Avg Context / Agent | DSpark |
Status | Peak VRAM (MiB) |
Aggregate Throughput (tok/s) |
Per-Agent (tok/s) |
Batch Wall Latency (s) |
|---|---|---|---|---|---|---|---|---|---|
Heretic-UD-Q4_K_M |
262,144 |
1 |
262,144 |
No |
OK | 21644 |
56.2 |
56.2 |
2.84 |
Heretic-UD-Q4_K_M |
262,144 |
2 |
131,072 |
No |
OK | 21680 |
81.99 |
40.99 |
3.9 |
Heretic-UD-Q4_K_M |
262,144 |
4 |
65,536 |
No |
OK | 21844 |
102.62 |
25.66 |
6.23 |
Heretic-UD-Q4_K_M |
262,144 |
8 |
32,768 |
No |
OK | 22198 |
145.14 |
18.14 |
8.81 |
Heretic-UD-Q4_K_M |
262,144 |
12 |
21,845 |
No |
OK | 22570 |
178.03 |
14.84 |
10.78 |
Heretic-UD-Q4_K_M |
262,144 |
16 |
16,384 |
No |
OK | 22902 |
197.57 |
12.35 |
12.94 |
Heretic-UD-Q4_K_M |
262,144 |
24 |
10,922 |
No |
OK | 23716 |
225.32 |
9.39 |
17.02 |
Heretic-UD-Q4_K_M |
262,144 |
1 |
262,144 |
Yes |
OK | 24146 |
54.83 |
54.83 |
2.91 |
Heretic-UD-Q2_K_XL |
524,288 |
8 |
65,536 |
No |
OK | 18440 |
154.11 |
19.26 |
8.3 |
Heretic-UD-Q2_K_XL |
524,288 |
16 |
32,768 |
No |
OK | 18984 |
208.88 |
13.05 |
12.25 |
Heretic-UD-Q2_K_XL |
524,288 |
24 |
21,845 |
No |
OK | 19616 |
236.14 |
9.84 |
16.25 |
Heretic-UD-Q2_K_XL |
524,288 |
32 |
16,384 |
No |
OK | 20100 |
247.48 |
7.73 |
20.67 |
Heretic-UD-Q2_K_XL |
524,288 |
40 |
13,107 |
No |
OK | 20942 |
243.53 |
6.09 |
26.25 |
Heretic-UD-Q2_K_XL |
524,288 |
48 |
10,922 |
No |
OK | 21606 |
241.84 |
5.04 |
31.72 |
Heretic-UD-Q2_K_XL |
1,048,576 |
8 |
131,072 |
No |
OOM | - |
- |
- |
- |
llama.cpp + Turbo-DSpark (GGUF)# 1. Apply the sm_86 + Banded Q4/Q8 KV + Turbo-DSpark patch to llama.cpp and build:
git apply patches/llama_inkling2_sm86.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
# 2A. Interactive / Single-Stream Reasoning with Turbo-DSpark (~74-84 tok/s):
./build/bin/llama-server \
-m Inkling-2-Mini-Heretic-UD-Q3_K_XL.gguf \
-md dspark-Inkling-2-Mini-Q8_0.gguf \
--spec-draft-n-max 3 \
-ngl 999 -ngld 999 -fa on \
-ctk q4_0 -ctv q4_0 -c 65536 --port 8080
# 2B. High-Throughput 32-to-48 Autonomous Agent Swarm (~247.5 tok/s aggregate, 512K shared pool on 24 GB):
./build/bin/llama-server \
-m Inkling-2-Mini-Heretic-UD-Q2_K_XL.gguf \
-ngl 999 -fa on -ctk q4_0 -ctv q4_0 -kvu \
-c 524288 -np 32 -b 256 -ub 256 --port 8080
Safetensors (JoaoZaokk/Inkling-2-Mini-Heretic)Use JoaoZaokk/Inkling-2-Mini-Heretic directly with Inkling2LoopForConditionalGeneration-compatible vLLM / Transformers runtimes (supports native NVFP4 + MXFP8 + mtp.safetensors DSpark speculative decoding) or convert custom GGUF quantizations via convert_hf_to_gguf.py.
thinkingmachines/Inkling-2-Mini).imatrix: Adapted from unsloth/Inkling-2-Mini-GGUF prior to dual-side Heretic subspace projection.thinkingmachines/Inkling-2-Mini