Mellum2.1-12B-A2.5B MaxQuality iMatrix GGUF

Quantized builds of JetBrains/Mellum2.1-12B-A2.5B-Thinking — a 12B-total / 2.5B-active MoE thinking model for agentic coding, math and reasoning. Calibrated iMatrix quants, measured on real hardware, tuned launch profiles.

⚡ Speed at a glance (8K context)

Why AllGPU wins here: Mellum is a MoE (64 experts, 8 active per token). Once any experts spill to CPU, PCIe round-trips dominate — same lesson as our other MoE releases: keep the whole model in VRAM or pay dearly. Q4_K_M at 7.5 GiB fits a 16 GB card outright with the full native context in KV cache; Q3_K_M at 5.9 GiB targets cards with less VRAM to spare.

Tested hardware and software

• Linux
• NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM
• AMD Ryzen 9 5950X, 16 cores / 32 threads
• 64 GB system RAM
• giveen/llama-cpp-turboquant, commit 8a891f4b (2026-07-31)

This is a MoE model (64 routed experts, 8 activated). On AllGPU profiles no -ot/--n-cpu-moe line is needed; the full-context profile below uses --n-cpu-moe to park deep experts in system RAM.

About this release

• Converted from the upstream bf16 checkpoint with the stock convert_hf_to_gguf.py; hybrid attention (SWA 1024 on 3 of every 4 layers, full attention every 4th) is handled natively by the converter — no architectural surgery.
• All files quantized with the same iMatrix pipeline as our other releases (calibration: imatrix-mellum21.dat). No per-tensor overrides — the stock K-quant presets do sensible work here.
• Thinking model: emits <think>…</think> before the final answer; use the --jinja flag and a qwen3-style reasoning parser (turboquant handles this out of the box).
• Recommended sampling (from upstream): temperature=0.6, top_p=0.95, top_k=20.

Files

Recommended launch commands

All three profiles run the whole model on the GPU (--n-cpu-moe 0 makes it explicit). TURBO_AUTO_ASYMMETRIC=0 disables turboquant's automatic asymmetric KV quantization so the symmetric turbo3/turbo4 cache types apply as specified.

131K context, single agent (recommended baseline)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Takes a bit over 10 GB of VRAM — the whole model plus a full native-context KV cache on one card. Generation speed settles around ~100 tok/s once the context grows to ~40K tokens; short-context bursts run faster (see the speed table above). Keep it simple: no speculative decoding flags needed — this model is fastest run plain.

Two agents at once (--parallel 2)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 262144 --parallel 2 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Same ~10 GB VRAM footprint, split into two 131K slots. With two agents working simultaneously at ~5K tokens of context each, each one still gets ~100 tok/s — the practical profile for running a coding agent pair (e.g. planner + executor) off one card.

Q3_K_M — 131K context, ~9 GB VRAM

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q3_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Fits cards with a bit under 9 GB of spare VRAM. ~130 tok/s at ~40K tokens of context — measurably faster than Q4_K_M at long context (fewer weight bytes to stream per token), and it holds the full native 131K window in VRAM.

Q3_K_M — for 6–8 GB VRAM cards (expert split CPU/GPU)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q3_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  -ot "blk\\.(1[4-9]|2[0-7])\\.ffn_.*_exps\\.weight=CPU" \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 8192 --ubatch-size 1024 --cache-reuse 2048

If your card only has 6–8 GB of VRAM, park the routed experts of the deep half (blocks 14–27) in system RAM with the -ot line while attention and the shallow layers stay on GPU. The smaller batch/ubatch and cache-reuse tune the split so prompt processing doesn't overflow the card.

256K context experiments (turbo4 KV)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja \
  --host 0.0.0.0 \
  --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 \
  --n-cpu-moe 0 \
  --ctx-size 262144 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo4 \
  --cache-type-v turbo4 \
  --batch-size 8192 \
  --ubatch-size 2048 \
  --cache-reuse 512

For experiments beyond the native 131,072 context — 262,144 tokens is 2× the model's native window, so treat output quality past 131K as uncharted territory. Uses the denser turbo4 KV cache and a smaller batch/ubatch to keep the larger cache affordable.

---

Upstream model card

This release is based on the original JetBrains/Mellum2.1-12B-A2.5B-Thinking model card, reproduced below. Original license: Apache 2.0.

Mellum2.1 is a thinking model. Use it for complex agentic tasks, such as working in a repository, running commands, and calling tools, and for hard non-agentic problems in coding, math, and reasoning.

Mellum2.1 Highlights

Mellum2.1 is the next version of Mellum2 Thinking. The architecture is unchanged (a 12B mixture-of-experts model with 2.5B active parameters), and almost all of the work for this version went into post-training, primarily reinforcement learning (RL):

• Reinforcement learning at a new scale: RL went from a short final stage to the main part of training, after many experiments on both the methods and the data.
• More data, filtered harder: new RL tasks in math, competitive programming, science, tool use, and software engineering, combining open RL datasets with tasks JetBrains built themselves and filtering every source before training.
• Real environments for agentic skills: for software engineering, the model trains inside real repositories with a shell and file-editing tools and is rewarded when the tests pass.

As a result, Mellum2.1 handles agentic tasks much better than Mellum2. After millions of sandboxed runs in real environments during training, it explores a codebase, edits files, and checks its own changes.

Model Overview

Mellum2.1 Thinking has the following features:

• Number of Parameters: 12B total, 2.5B active
• Number of Layers: 28
• Hidden Size: 2304
• Intermediate Size: 7168
• MoE Intermediate Size: 896
• Number of Experts: 64
• Number of Activated Experts: 8
• Number of Attention Heads (GQA): 32 for Q and 4 for KV
• Context Length: 131,072
• Sliding Window: 1,024 (3 of every 4 layers)
• Vocabulary Size: 98,304
• Precision: bfloat16
• License: Apache 2.0

Evaluation

All values are percentages; higher is better except HarmBench, where lower is better. All models were evaluated by JetBrains with the same pipeline in thinking mode. All values are self-reported by JetBrains.

Notes:

• Non-agentic benchmarks use greedy decoding.
• Agentic benchmarks use the same open-source agent harness (Pi v0.73.1, shell and file tools) for every model, with each model's default sampling (temperature 1.0 for Mellum2.1), a 114K-token context, and up to 16K tokens per turn.
• AIME is the mean of AIME 2025 and AIME 2026 (30 questions each).
• BFCL v4 is the macro-average of five subtasks: v1, v2, v3, web search, memory.

Quickstart (upstream, Transformers/vLLM)

from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Find the bug in this function and explain the fix: def mean(xs): return sum(xs) / len(xs) - 1"},
]

chat_response = client.chat.completions.create(
    model="JetBrains/Mellum2.1-12B-A2.5B-Thinking",
    messages=messages,
    max_tokens=81920,
    temperature=0.6,
    top_p=0.95,
    extra_body={"top_k": 20},
)
print("Chat response:", chat_response)

License

Released under the Apache 2.0 license (inherited from the upstream model). The upstream model card and its license declaration are preserved unchanged above.