Mellum2.1-12B-A2.5B-Thinking — Q3 / Q4K / Q5K GGUFs
Fast GGUF for mainstream llama.cpp, quantized antirez style: routed experts quantized hard, everything else protected.
Recipe (antirez style)
Down-expert note: Mellum's down-expert rows are 896 wide and K-quants require
multiples of 256, so K-quant downs are impossible there (Q5_0 is llama.cpp's
standard fallback — same as bartowski's Q4_K_M layouts; the Q3_K down
falls back to Q4_0). Imatrix:
bartowski calibration v6.
| Tensor group | Q3 | Q4K | Q5K | Q5K-Q8 |
|---|---|---|---|---|
| Routed expert gate / up | IQ3_XXS | Q4_K | Q5_K | Q5_K |
| Routed expert down | Q4_0 | Q5_0 | Q5_0 | Q8_0 |
| Shared experts, attention, output | Q8_0 | Q8_0 | Q8_0 | Q8_0 |
| Token embeddings | F16 | F16 | F16 | F16 |
| Routers, norms, biases | F32 | F32 | F32 | F32 |
| File | Size | BPW | KL ↓ | Same-top ↑ | q36 hard-smoke |
|---|---|---|---|---|---|
| Mellum2.1-12B-A2.5B-Thinking-Q3-IQ3_XXS-Q4_0.gguf | 5.83 GiB | 4.12 | 0.164 | 83.6% | 9/12 |
| Mellum2.1-12B-A2.5B-Thinking-Q4K-Q5_0.gguf | 7.49 GiB | 5.30 | 0.048 | 96.6% | 9/12 |
| Mellum2.1-12B-A2.5B-Thinking-Q5K-Q5_0.gguf | 8.36 GiB | 5.91 | 0.0239 | 99.1% | 9/12 |
| Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf | 9.65 GiB | 6.82 | 0.0135 | 100% | 10/12 |
KL vs Q8_0 reference logits (same prompts, 2 chunks @ ctx 60, CPU). Same
protocol put bartowski's Q5_K_M at 0.0165 (9.28 GiB), his Q6_K_S at 0.0123
(10.37 GiB) and wepiqx MERNIK-10000 at 0.0361 (9.46 GiB).
q36 hard-smoke: 12 reasoning/reading-comprehension cases from the q36 engine test set, temp 0, llama.cpp Vulkan vs the same reference runs. For scale: LFM2.5-24B-A2B-Q3 8/12, gemma-4-26B Q2 7/12, G9v3 Q2 5/12, Ling-3.0-tiny Q4 4/12.
Usage
# interactive chat
llama-cli -m Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf
# server
llama-server -m Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf --host 0.0.0.0 --port 8080
Model
Mellum2.1-12B-A2.5B-Thinking is a 64-expert MoE (12 B total, 2.5 B active) for code reasoning. Context: 128k tokens.
Original model: JetBrains/Mellum2.1-12B-A2.5B-Thinking. Original BF16 GGUF: bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF. Quantized from BF16 with a streaming quantizer (bounded RAM/disk window, network as bottleneck); the BF16 was never stored in full. License: Apache-2.0.