Mellum2.1-12B-A2.5B-Thinking — MERNIK quants

STATUS: all builds live — 7500, 10000, 6500 pair. Pick your trade.

Base: JetBrains/Mellum2.1-12B-A2.5B-Thinking (12B MoE, 2.5B active, 28 layers, 64 experts × 8 active). Method: MERNIK measure-first allocation (smse utility) on the vendor imatrix. MoE routers (ffn_gate_inp) pinned out-of-budget on all 28 layers — the allocator's own rule holds them outside the queue, and the binary lays F32 (measured 28/28 F32 in every file, including stock — same class of behavior as the 1D law).

Honest standings (our protocol, temp 1.0)

Duel 10000-SMSE vs Q6_K: 3/3 SIGNIFICANT (HE 23/8 p=0.0107, HE+ 22/8 p=0.0161, empties 5/20 p=0.0041) — the first statistically real quant-vs-stock win of the campaign. The +15 clears the 13-task instrument floor. Duel 7500-SMSE vs Q6_K: 0/3, NOISE (HE 18/12 p=0.3616, HE+ 18/12 p=0.3616, empties 13/17 p=0.5847 — all three columns sub-floor). The +6 lead is a lead, not a crown — stated as lead. The weight saving (−3.4 GB) is arithmetic, not statistics. Duel 6500-SMSE vs 6500-RECOVERY: 0/3, NOISE (HE 10/7 p=0.6291, HE+ 10/6 p=0.4545, empties 6/11 p=0.3323). Note the canary/verdict split: recovery owns PPL by −0.48 (10.70 vs 11.18) yet ties on all three verdict columns — the canary demoted, once more. Empties: 6500-SMSE 33/164, 6500-recovery 38/164 — same silence ladder (10000: 24, 7500: 35, Q4_K_M: 33, Q6: 39). Note the oddity: stock Q4 goes silent less than stock Q6 (33 vs 39) while also scoring higher (125 vs 119) — but the duel is 0/3 NOISE, so the ladder wobbles within sampling. Silence is signal, just a quiet one. Note: the Q4 floor sits at 7428 MiB — 7500 is the minimum viable build without sub-Q4 tiers; 5500 proved impossible (floor 5688 MiB even with Q3 allowed).

Notes

• 10000 build vs the 1070. Two HE attempts died at [54/164] and [90/164] with ggml-cuda.cu:109 CUDA error (kernel fault under factory OC, not OOM), a third died at [150/164] to a RAM-OOM (63 GB VM — parallel slots × 8K context). Cures that held: stock clocks, --parallel 1, --cache-reuse 64. The closed battery ran on all three.
• Hardware: GTX 1070 8 GB, partial offload (-ngl 20–28), consumer rig, not a datacenter.
• Noise floor: at n=164, gaps below ~13 tasks do not reproduce run to run. Temp-0 makes the verdict column reproducible (measured: 164/164 identical on re-run).
• Silence scales with weakness: empty completions — 10000: 24/164, 7500: 35/164, stock Q6: 39/164. Weaker builds go silent rather than babble; the empties column separated the 10000-vs-Q6 duel at p=0.0041 all by itself.

Protocol (so anyone can re-check)

• Scores: llama-server -ngl {18,20,28} -c 8192 --jinja, sampling temp 1.0 / top_p 0.95 / top_k 20 / max_tokens 2048, strictly sequential, fixed seed 7 for PPL (-c 1024 -n 64 -b 512, wikitext-2-raw).
• HE+ via evalplus.evaluate. Three columns or it stays unverified.

Speed tip (interactive use, NOT the benchmark)

For serving (not scoring), ngram speculation lifts codegen from ~60 t/s to ~160 t/s on this model:

llama-server -m MODEL.gguf -ngl 99 -c 8192 --jinja \
  --spec-type ngram-map-k4v --spec-ngram-map-k4v-size-m 12 --parallel 1

All batteries above were scored WITHOUT speculation (strictly sequential, same protocol as every other build in the lab) — speed tricks never enter the verdict column.

Reproduce

python main.py --model Mellum2.1-12B-A2.5B-Thinking-BF16.gguf \
  --imatrix Mellum2.1-12B-A2.5B-Thinking-imatrix.gguf \
  --size 7500 --utility smse --output Mellum2.1-12B-A2.5B-Thinking-MERNIK-7500-SMSE.gguf --run

---

If this work saved you a GPU-week: github.com/sponsors/wepiqx