Mellum2-12B-A2.5B-Thinking GGUF (Q4_K_M)

Quantized GGUF version of JetBrains/Mellum2-12B-A2.5B-Thinking for use with llama.cpp.

⚠️ Requires llama.cpp with Mellum architecture support. This is not yet in mainline llama.cpp — use PR #23966 or a build that includes it.

Model Details

| Property | Value |

|----------|-------|

| Base model | JetBrains/Mellum2-12B-A2.5B-Thinking |

| Architecture | Mellum (MoE) |

| Total parameters | 12B |

| Active parameters | 2.5B |

| Experts | 64 (8 per token) |

| Context length | 128K (sliding window) |

| Quantization | Q4_K_M |

| File size | ~7.5 GB |

Usage

llama-server \
  -m Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf \
  -c 32000 \
  -ngl 99 \
  --jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --port 18081

For thinking mode, use reasoning_budget to cap reasoning tokens:

curl http://localhost:18081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Mellum2-12B-A2.5B-Thinking-Q4_K_M","messages":[{"role":"user","content":"Explain quicksort"}],"max_tokens":500,"reasoning_budget":512}'

Low VRAM (6GB GPU)

For GPUs with limited VRAM, offload MoE experts to CPU:

llama-server \
  -m Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf \
  -c 32000 \
  -ngl 99 \
  -ncmoe 20 \
  -t 8 \
  --jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0

Tested on GTX 1060 6GB: ~18 t/s with ncmoe=20, ~22 t/s with ncmoe=10.

Acknowledgements

• JetBrains for the original Mellum2 model
• llama.cpp for the inference engine