Mellum 2.1 12B-A2.5B Thinking, selective precision: attention Q8_0, experts Q5_K (GGUF)
Release: built to be downloaded and served with Xyntetik Runner.
- Measured status against the parent: passes the adopted bar (margin-qualified top-1 >= 97% and mean KLD <= 0.05) against JetBrains' model in float32, where the publisher's own Q4_K_M fails it; the plan and the numbers are on this card.
- Parent model: JetBrains/Mellum2.1-12B-A2.5B-Thinking at
92ddae9fc766, through a BF16 GGUF conversion of that checkpoint.- Collection: Runner Releases, also in Pruning and Quantization Frontiers
- Evidence: the fidelity tables on this card, measured at 300 teacher-forced positions; the full record is in the Runner repository.
- Runner compatibility: served by Runner 1.13.0 and later. Written with
--rewrite --type-planby Runner at commit51c766d(the q5_k, q5_1 and iq4_nl writers, on Runner main since PR #632 and in the next release).
- Validation status (Runner support tiers): each file's row in the Runner roster says which tier it is in, what is re-checked on a release and the date and version of the last run.
Mellum2.1-12B-A2.5B-Thinking-selective-q8_0-expq5_k.gguf: roster row (Tier 2 since 2026-11-06: tokenizer 0 of 721 strings and 0 of 97,302 vocabulary entries against JetBrains' tokenizer, template conformance 23 of 23, CPU and CUDA replies byte-identical).
Want this for another model? The tools behind these files are open: requantization, expert pruning and sublayer removal in Xyntetik Runner, and the method for this file is on this card or in its evidence dataset.
Request a model · Report your result · Community results · All files
If this file is useful to you, a contribution funds the hardware time behind the next one.
Serve with xyntetik-runner — tool calls that still parse when the budget runs out
This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:
When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.
Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models — tight context, slow generation — that is the difference between an agent loop that finishes and one that retries from scratch.
Measured — what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1→64):
| engine | budget too small (1–16 tokens) | enough budget (64, control) |
|---|---|---|
| Runner | executable tool_calls, arguments parse |
completes |
| vLLM 0.27.1 | no call; protocol framing leaks into content |
completes |
| llama.cpp b10488 | no call; leak, then tool_calls with unparseable args |
completes |
| Ollama 0.32.14 | no call; empty content, then HTTP 500 | completes |
| TensorRT-LLM 1.2.1 † | no call; <tool_call> leak, then empty content |
completes |
| SGLang 0.5.17 † | no call; <tool_call> leak, then empty content |
completes |
Every engine completes at 64 tokens — the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. † TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.
Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) — it guarantees the shape of a call at any quantization, not its contents.
Quickstart
# build the single binary (no deps), then serve this GGUF:
runner -m Qwen3-30B-A3B-expq4_0-attnq8_0.gguf --serve
# point any OpenAI-compatible client at http://localhost:8080/v1
Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.
What this file is, and is not
Mellum2.1-12B-A2.5B-Thinking-selective-q8_0-expq5_k.gguf is an 8.99 GB GGUF of JetBrains' Mellum 2.1 12B-A2.5B Thinking, a mixture-of-experts code model, written by Xyntetik Runner's own rewriter from a BF16 conversion of the publisher's checkpoint. It is not the publisher's model, only measured against it, and it was measured on one text only: the project's gold corpus (gold-corpus-2k.txt: English technical documentation with code, English prose and one Swedish passage), 300 positions. No task benchmark was run, and no speed was measured.
Weakest numbers first:
- Plain top-1 agreement with the publisher's float32 model is 94.00%: at about one position in 17 the file's most likely next token differs from the reference's.
- Its first window (positions 8 to 749) reads a mean KLD of 0.0419, the closest of its three windows to the 0.05 bar.
- It is 31.4 MB larger than bartowski's Q5_K_S of the same model, and its mean KLD is 0.0004 higher. That file is the better choice if size is all that matters; this one is the recipe Runner writes itself, with no importance matrix.
- It is under 9 GB by 5.2 MB. An 8 GB machine does not hold it.
- Its three new tensor types (q5_k, q5_1 for the 896-wide expert down projections) were written without an importance matrix.
This file passes the bar: mean KLD 0.0293 against the publisher's float32 model (bar 0.05) and margin-qualified top-1 98.74% (bar 97%). The kill was declared before the read: missing either number ended the claim, with no retuning.
Against other files in the same window
Each file measured against the same reference at the same 300 positions.
| File | Bytes | Mean KLD | Median | Max | Top-1 | Margin-qualified top-1 | Bar |
|---|---|---|---|---|---|---|---|
| This file | 8,994,830,528 | 0.0293 | 0.0126 | 0.437 | 94.00% | 98.74% | pass |
| bartowski Q5_K_S (bartowski/JetBrains_Mellum2.1-12B-A2.5B-Thinking-GGUF) | 8,963,422,944 | 0.0289 | 0.0126 | 0.476 | 93.00% | 98.33% | pass |
| Publisher Q4_K_M (JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF) | 8,071,295,264 | 0.1172 | 0.0476 | 3.663 | 88.67% | 92.89% | fail |
By depth (mean KLD / margin-qualified top-1; 50, 83 and 167 positions):
| File | 8 to 749 | 750 to 1,999 | 2,000 to 4,493 |
|---|---|---|---|
| This file | 0.0419 / 100% | 0.0325 / 100% | 0.0239 / 97.8% |
| bartowski Q5_K_S | 0.0497 / 97.4% | 0.0282 / 100% | 0.0230 / 97.8% |
What the bytes bought
| File | Attention q | Attention k, v, output | Expert gate / up | Expert down (896 wide) | token_embd / output |
Routers |
|---|---|---|---|---|---|---|
| This file | Q8_0 (28) | Q8_0 | Q5_K / Q5_K | Q5_1 | Q8_0 / Q8_0 | F32 |
| bartowski Q5_K_S | Q8_0 (17), Q6_K (6), Q5_K (5) | Q8_0 | Q5_K / Q5_K | Q5_1 | Q8_0 / Q8_0 | F32 |
The two files differ in the attention q projections, and bartowski's was made with an importance matrix. The 896-wide expert down projections cannot take Q5_K's 256-wide block, so Runner writes them as Q5_1, its 32-block cousin (28 tensors, reported by the rewrite).
Recipe
From the publisher's checkpoint converted to a BF16 GGUF (parent-bf16.gguf, 24,311,969,984 bytes, sha256 8053a907bde4b833..., llama.cpp convert_hf_to_gguf.py at 73a43d1):
runner -m parent-bf16.gguf --rewrite Mellum2.1-12B-A2.5B-Thinking-selective-q8_0-expq5_k.gguf --type-plan plan.json
with plan.json:
{"default": "q8_0", "rules": [{"match": "ffn_gate_inp", "type": "keep"}, {"match": "_exps.weight", "type": "q5_k"}]}
Every one of the parent's 339 tensors is in the file, each at the plan's type. The header declares architecture mellum, apache-2.0, 28 blocks, 64 experts and 8 used. Two header values to know about:
general.namereadsCkpt Mem: the converter took it from the directory it read the checkpoint from, not from the model. It is left as written, because changing it changes the file's bytes and the sha256 the record pins.general.file_typeis 7 (Q8_0): the rewrite labels a mixed file by its most common quantized type. The per-tensor table above is the truth.
Fidelity
| reference | positions | mean KLD | median KLD | max KLD | top-1 | margin-qualified top-1 |
|---|---|---|---|---|---|---|
| publisher, float32 | 300 | 0.0293 | 0.0126 | 0.437 | 94.00% | 98.74% of 239 |
KL is over the reference's 20 most likely tokens, both sides renormalised; a position is margin-qualified when the reference's top two are more than 0.5 nats apart. Positions 8 + n x 15 for n < 300 of gold-corpus-2k.txt, token-id keyed, no special prefix.
Reference
The publisher's model JetBrains/Mellum2.1-12B-A2.5B-Thinking at 92ddae9fc766, transformers' own mellum code in float32 arithmetic (transformers 5.16.1, scripts/gold-logits.py). The bar (mean KLD at most 0.05, margin-qualified top-1 at least 97%) is stated against this reference only.
Run it
./runner -hf Joakimpalm-Zen/Mellum2.1-12B-A2.5B-Thinking-selective-Q8_0-expQ5_K-GGUF --serve
The file is 8.99 GB; the context comes on top. It is a thinking model, and Runner serves its template as the publisher ships it.
License
apache-2.0, as the parent's model card and the file's header state it.
Scope and limits
- One window over one corpus. Not a benchmark claim, not a speed row, not a claim about other text.
- The roster row's checks (tokenizer differential, template conformance, pinned replies) ran on the lab's CPUs and a CUDA MIG slice; no Mac or Windows run of this file exists yet.
- The file was written on a Linux CUDA host and measured there; no Mac or Windows run of this file exists yet.
Records
Mellum2.1-12B-A2.5B-Thinking-selective-q8_0-expq5_k.gguf.quant.json: the rewrite's own provenance record, as written, naming the base file, the type plan (R1.json, hereplan.json) and the output (R1.gguf, its name while it was made) by sha256, and the Runner binary that wrote it. The sha256 is the binding; the names are not.plan.json: the type plan, as given to--type-plan.SHA256SUMS: the file and the two records.- The measurement, the kill and the comparison files are in the Runner repository: docs/compat-reports/mellum2.1-rewrite-q5-2026-11-05.