Mellum2.1-12B-A2.5B-Thinking — FP8
Quality benchmark: Mellum2.1-12B-A2.5B-Thinking — FP8 (dynamic scales) vs BF16
TL;DR
- Uncapped run (t=0, up to 98k thinking tokens, 2,420 paired items, 6 benchmarks): BF16 80.6% vs FP8 81.5% (Δ +0.95 pp). Pairwise: BF16-only solves 83, FP8-only solves 106.
- No benchmark shows a significant FP8 deficit. FP8 is ahead on five of six (GSM8K +1.0, MATH-500 +0.6, MMLU-Pro +1.0, TriviaQA +1.7, AIME 2022–24 +1.1); AIME 2025 is −3.3 pp = exactly one problem.
- The one AIME 2025 loss is a non-convergence case, not an error: on problem 29 FP8 burned the full 98k budget without terminating, while BF16 converged at 46k tokens. Every other benchmark has maxlen hits at or below BF16's.
- AIME 2025 pass@8 (sampled, n=8, 98k): standard pass@8 96.7% (BF16) vs 93.3% (FP8) — 29/30 vs 28/30 problems solved at least once; consistency@8 (all 8 correct) 60.0% vs 73.3%; maj@8 90.0% vs 93.3%.
- Earlier apparent drops were budget artifacts, not quantization damage. Under a 32k cap the same quant once measured −8.9 pp on AIME 2022–24; with the cap removed, that suite is +1.1 pp in FP8's favor.
- Long context unaffected: NIAH 32k/64k — BF16 6/8, FP8 8/8.
Budget sensitivity (why caps matter for this checkpoint pair)
Measured accuracy as the thinking budget grows — same checkpoints, same prompts, greedy:
| Suite | FP8 @ 8k cap (static) | FP8 @ 32k cap (dynamic) | FP8 uncapped (dynamic) | BF16 uncapped |
|---|---|---|---|---|
| MATH-500 | 78.6% (105/500 maxlen) | 86.0% (3/500) | 86.2% (1/500) | 85.6% (3/500) |
| AIME 2022–24 | 54.4% (48/90) | 73.3% (14/90) | 86.7% (2/90) | 85.6% (1/90) |
| AIME 2025 | 50.0% (16/30) | 80.0% (6/30) | 86.7% (1/30) | 90.0% (0/30) |
Two effects were confounded in early measurements: (a) static per-tensor activation scales caused greedy repetition loops that burned the budget — the dynamic per-token scales build eliminated the loops entirely (0 verbatim loops at 32k vs 70/500 at 8k); (b) the cap itself turned slow-but-correct trajectories into failures. Uncapping the budget and switching to dynamic scales resolves both; the residual difference between the checkpoints is within noise everywhere except one AIME 2025 non-convergence.
Methodology
- Endpoints: two vLLM OpenAI-compatible servers serving the original (BF16) vs this checkpoint (FP8), identical prompts; greedy t=0 for the deterministic suite, card-protocol sampling (t=0.6, top_p=0.95, top_k=20, fixed seeds, n=8) for AIME pass@k.
- Thinking budgets:
max_tokens=98000for MATH-500 / AIME greedy and sampled (servermax_model_len=100000; remaining maxlen hits are genuine non-convergence, not budget squeeze — uncapped AIME trajectories >32k occur on 5/30 items per checkpoint), 16,384 GSM8K, 32,768 MMLU-Pro, 8,192 TriviaQA. Maxlen rates are low and symmetric (see table), so caps don't drive the comparison. - Scoring: GSM8K by final number; MATH-500/AIME by last
\boxed{}with LaTeX normalization (leading zeros normalized:080==80); MMLU-Pro byANSWER: Xletter; TriviaQA by last-line alias match; HumanEval by sandboxed execution with the official grader; NIAH by needle number. - Statistics: exact McNemar on paired items; AIME pass@k reported both as the standard unbiased estimator (≥1 of k correct) and as consistency@k (all k correct).
- HumanEval (exec) numbers are from the 4k-budget run: BF16 72.0% vs FP8 76.2% (+4.3 pp) on 164 items, 16/164 vs 18/164 cap hits — nearly symmetric, budget not the differentiator.
Deterministic suite (greedy, t=0, uncapped)
| Benchmark | n | BF16 | FP8 | Δ (pp) | McNemar p | max-len hits B/F |
|---|---|---|---|---|---|---|
| GSM8K | 500 | 94.0% | 95.0% | +1.00 | 0.442 | 0 / 1 |
| MATH-500 | 500 | 85.6% | 86.2% | +0.60 | 0.678 | 3 / 1 |
| AIME 2025 | 30 | 90.0% | 86.7% | -3.33 | 1 | 0 / 1 |
| AIME 2022–24 | 90 | 85.6% | 86.7% | +1.11 | 1 | 1 / 2 |
| MMLU-Pro (sample) | 1000 | 75.2% | 76.2% | +1.00 | 0.363 | 14 / 15 |
| TriviaQA (sample) | 300 | 65.3% | 67.0% | +1.67 | 0.487 | 11 / 14 |
| HumanEval (exec) | 164 | 72.0% | 76.2% | +4.27 | 0.34 | — |
| NIAH 32k + 64k | 8 | 75.0% | 100.0% | +25.0 | 1 | 0 / 0 |
Sampled (card protocol: t=0.6, top_p=0.95, top_k=20, n=8, 98k budget, AIME 2025)
| Metric | BF16 | FP8 | Δ |
|---|---|---|---|
| pass@8 (≥1 of 8, unbiased) | 96.7% | 93.3% | -3.3 |
| consistency@8 (all 8 correct) | 60.0% | 73.3% | +13.3 |
| maj@8 | 90.0% | 93.3% | +3.3 |
| avg over 8 samples | 87.1% | 87.9% | +0.8 |
| median tokens / sample | 9,785 | 10,680 | +9% |
The pass@8 gap is one problem (29 vs 28 of 30 solved at least once); consistency@8 favors FP8 by 13 pp — when FP8 is right, it is right more stably across samples.
Where the two checkpoints diverge (greedy, uncapped)
Discordant items: gsm8k 27 (B+11/F+16), math500 23 (B+10/F+13), aime25 1 (B+1/F+0), aime2224 7 (B+3/F+4), mmlupro 98 (B+44/F+54), triviaqa 33 (B+14/F+19) — FP8 net-positive everywhere except the single AIME 2025 item.
- The one AIME 2025 FP8 loss (
aime25-29): non-convergence — 98k tokens, no\boxed{}; BF16 solved it in 46,302 tokens. AIME 2025 items with >32k-token trajectories: 5 on each side, so neither checkpoint is uniquely prone to marathon reasoning here. - Thinking lengths are converged: MATH-500 paired medians 1,111 vs 1,082 tok (FP8 longer in 48% of pairs), GSM8K 489 vs 466 (46%); vs the static-scale build, which inflated lengths +29% and looped to the cap on 70/500 MATH-500 items.
Token-level drift vs BF16
Raw-text corpus (EN: prose / Python / math), prompt_logprobs=20, 46 segments; 39,734 next-token positions:
| Segment | positions | top-1 agree | JSD (top-20) |
|---|---|---|---|
| overall | 39,734 | 90.0% | 0.0275 |
| prose_en | 22,640 | 86.9% | 0.0346 |
| code_en | 13,691 | 94.0% | 0.0189 |
| math_en | 3,403 | 95.2% | 0.0143 |
Measured against the static-scale build; the dynamic build's behavioral drift (loop rates, thinking lengths, benchmark deltas) matches BF16, but the logprob-level numbers above predate the rebuild. Actual-token NLL deltas (+0.39) are typical of W8A8 quants calibrated on short static samples.
Throughput note
| Endpoint | TTFT (8 tok) | conc=1 | conc=8 | conc=32 |
|---|---|---|---|---|
| bf16 | 0.17s | 207 | 1289 | 3506 |
| fp8 | 0.22s | 244 | 2085 | 5207 |
Decoded tok/s at 512-token greedy completions, measured isolated after all quality stages. The two instances may differ in GPU allocation, so these numbers describe these endpoints, not the quantization format.
Disclaimer
This is not the original model. This repository contains only an FP8 quantization of JetBrains/Mellum2.1-12B-A2.5B-Thinking.
- I am not the developer or owner of this model.
- All rights to the model belong to JetBrains and the creators of the original model.
- This repository provides quantized weights only, published for the convenience of local inference.
About the original model
Mellum2.1-12B-A2.5B-Thinking is a thinking model for coding, math, and agentic tasks, built on a mixture-of-experts architecture:
- 12B total parameters, 2.5B active per token (MoE: 64 experts, 8 activated per token)
- 28 layers, hidden size 2304, GQA (32 Q heads / 4 KV heads)
- Context length up to 131,072 tokens (sliding window 1,024 on 3 of every 4 layers)
- Trained primarily with reinforcement learning, including agent training in real repositories with shell and file-editing tools
- Particularly strong on LiveCodeBench, SWE-bench, and agentic tool-coding tasks
For architecture details, training setup, and official benchmark results, see the original model card.
Quantization details
Weights of all Linear layers are quantized to FP8 (E4M3, symmetric) with the compressed-tensors format (vLLM and Transformers both run it natively — no trust_remote_code and no calibration plugins needed):
| Target | Precision |
|---|---|
All Linear weights (attention projections, MoE experts) |
FP8-E4M3, symmetric, per-channel scales (strategy channel) |
| Input activations | FP8-E4M3, symmetric, dynamic per-token quantization at runtime (strategy token, dynamic: true) — no calibration data needed |
MoE router gates (mlp.gate) |
kept in bfloat16 — routers are sensitive to quantization |
lm_head |
kept in bfloat16 |
| KV cache | kept in bfloat16 (kv_cache_scheme is null) |
| Embeddings, norms | kept in bfloat16 |
Unquantized layers are listed in quantization_config.ignore in config.json.
Quantized with llm-compressor/compressed-tensors 0.18.0.
Usage
The model is used exactly like the original — just point model_id at this repository.
FP8 requires an NVIDIA GPU with compute capability 8.9+ (Ada Lovelace, Hopper, Blackwell).
Transformers
pip install transformers accelerate torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "voves/Mellum2.1-12B-A2.5B-Thinking-FP8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
)
messages = [
{"role": "user", "content": "Find the bug in this function and explain the fix: def mean(xs): return sum(xs) / len(xs) - 1"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output_ids = model.generate(
**inputs,
max_new_tokens=32768,
temperature=0.6,
top_p=0.95,
top_k=20,
)
print(tokenizer.decode(output_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=False))
vLLM
vllm serve voves/Mellum2.1-12B-A2.5B-Thinking-FP8 \
--max-model-len 131072 \
--reasoning-parser qwen3
With tool calling:
vllm serve voves/Mellum2.1-12B-A2.5B-Thinking-FP8 \
--max-model-len 131072 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes
After the server starts, send a request:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "voves/Mellum2.1-12B-A2.5B-Thinking-FP8",
"messages": [
{"role": "user", "content": "Find the bug in this function and explain the fix: def mean(xs): return sum(xs) / len(xs) - 1"}
],
"max_tokens": 32768,
"temperature": 0.6,
"top_p": 0.95
}'