Mellum2.1-12B-A2.5B-Thinking: APEX GGUF

These are measured-allocation APEX quants of JetBrains/Mellum2.1-12B-A2.5B-Thinking.

The base model is a 12.15B-parameter MoE, with about 2.44B parameters active per token:

• 28 layers, 64 routed experts, top-8 routing, and no shared expert;
• grouped-query attention (32 query and 4 KV heads, head dim 128);
• a sliding-window pattern of three 1024-token SWA layers to one full-attention layer.

It has a reasoning channel.

The four tiers run from 4.9 down to 3.1 bits per weight. Each has a KL-divergence measurement against BF16, and each was run twice through a 42-task agentic coding suite.

Files

Also in this repo:

• Mellum2.1-12B-A2.5B-Thinking-codeweighted.imatrix, the importance matrix used for every file.
• recipes/*.recipe, the exact per-tensor type assignment for each tier, for use with llama-quantize --tensor-type-file.
• sensitivity/Mellum2.1-12B-A2.5B-Thinking.sensitivity.json, the measured per-tensor KL map the allocation was built from. Absolute values are specific to the GPU that measured them (a DGX Spark); only the rankings transfer.

Which to pick:

• 4.9 bpw if it fits. It beats a stock Q4_K_M on every KLD column while being 8% smaller.
• 3.7 bpw for a tight memory budget. It showed no measured agentic cost against the larger files.
• 3.1 bpw is the floor. It works, but it is measurably worse in the agentic runs (see below).

Usage (llama.cpp)

llama-server -m Mellum2.1-12B-A2.5B-Thinking-APEX-4.9bpw.gguf -ngl 99 --jinja \
  -c 131072 --flash-attn on --temp 1.0 --top-p 0.95 --top-k 20

• Needs a llama.cpp build with the mellum architecture. Every number here was produced on mainline 6d0549831; no fork is needed.
• --jinja is required for the chat template. Reasoning comes back in reasoning_content, and tool calls work.
• The KV cache is small: 14 KiB per token, plus a constant 42 MiB for the SWA layers. The full 128k context costs about 1.75 GiB, so the 4.9 bpw file at 128k fits on a 16 GB card.
• Decode speed on an RTX 4060 Ti 16 GB was 126, 130, 140 and 153 tok/s for 4.9, 4.3, 3.7 and 3.1 bpw.

How these were made

• Measured allocation, not a fixed band recipe. The bit allocation comes from dynamic_apex.py with a per-tensor sensitivity map, measured by quantizing each tensor alone and recording the KL divergence against BF16. The tensor budget then goes where it costs the least KL. This differs from the structural "edge/near/middle" depth bands of earlier APEX quants.
• The expert down-projections cannot use k-quants. ffn_down_exps rows are 896 wide, and 896 is not a multiple of 256, so every QK_K type is illegal there. The legal rungs are Q8_0, Q5_0 and IQ4_NL (block 32) and Q2_0 (block 64). There is no 3-bit option, so the step below IQ4_NL is 4.5 → 2.25 bpw.
  • This is also why a stock "Q4_K_M" of this model is really 5.31 bpw: its down-projections fall back to Q8_0 and Q5_0.
• Imatrix-aware Q2_0. At 3.7 and 3.1 bpw the allocator put Q2_0 on 6 and 22 of the 28 down-projections.

  • Stock Q2_0 ignores the imatrix (plain round-to-nearest). These files were encoded with a patched quantizer that refits Q2_0 under the imatrix weights.

  • The patch changes encoding only. The format is the standard Q2_0, and it decodes on stock llama.cpp.

  • Same-size control arms that kept every down-projection at IQ4_NL instead:

    size	measured (Q2_0 down)	control (IQ4_NL down)
    3.7 bpw	KLD 0.208	KLD 0.246
    3.1 bpw	KLD 0.484	KLD 0.517

    The measured allocation wins at both sizes.

• Imatrix. The imatrix came from a code-weighted calibration corpus, about 70% code.
• KLD. KLD was measured with llama-perplexity --kl-divergence: 100 chunks of 512 tokens from a code-heavy evaluation text, separate from the calibration corpus. The BF16 perplexity on the same text is 1.672.

Agentic benchmark

The suite has 42 coding tasks: 25 general, 5 hard, 9 frontier and 3 long-horizon. A model works in OpenCode 2.0.26 (real agentic loop, tools, its own test runs) and is graded by a fresh pytest it never saw.

Settings: 128k context, 32k output cap, temp 1.0, top-p 0.95, on an RTX 4060 Ti 16 GB. Each file ran twice. bartowski's Q6_K_L and the stock Q4_K_M ran as reference points.

How to read it:

• This suite is built so that a competent model scores near 42/42. A full score is the entry bar, and the comparison that discriminates is output tokens for the same passing result.
  • One task, a backtracking regex engine, flips between runs for every file (Q6_K_L scored 16/59 tests, then 0/59). It accounts for most of the misses.
• 3.1 bpw is a real regression.
  • It needs about 2.2x the output tokens of 4.9 bpw for the same passing work, in both runs and on 29 of 33 tasks.
  • It had the most failures.
  • In its first run, its solution to one task consumed enough memory to halt the suite. That task and the four after it were scored as failed.
• The other differences are not resolved at two runs. The token gaps between Q6_K_L, 4.9 bpw, Q4_K_M and 3.7 bpw are within run-to-run spread. Their direction agrees with KLD.
• 4.3 bpw failed the same task (a B-tree insert/delete) in both runs, where every other file passed it. Treat that as unconfirmed until it has more runs.
• Code quality is not measured. The suite records tests passed and tokens only.

Attribution

• Base model: JetBrains, JetBrains/Mellum2.1-12B-A2.5B-Thinking.
• The APEX recipe lineage: LocalAI, localai-org/apex-quant.
• Quantization engine: llama.cpp (ggml-org).

This is an unofficial community quantization, not affiliated with or endorsed by JetBrains.