Mellum2.1-12B-A2.5B-Thinking — MXFP4
Weight-only MXFP4 quantization of JetBrains/Mellum2.1-12B-A2.5B-Thinking, with a standalone Python runner tested on native Windows at batch size 1.
The checkpoint is one model.safetensors file: 6.638 GiB / 7,127,649,016 bytes. Weights use packed E2M1 FP4 values with E8M0 scales per 32 elements. The output head, embeddings, routers and normalization weights remain BF16. The runtime uses BF16 activations and KV.
Local measurements on an RTX 5070 Ti Laptop GPU, 12 GB reached approximately 111 tokens/s at short context and 85 tokens/s after a 32,000-token prompt, using FA2, static 32K KV, TorchInductor and explicit CUDA graphs. These are inference-speed measurements, not model-quality scores or guarantees for other hardware.
Run the model
The included Python runtime handles both single-turn generation and persistent multi-turn chat. It requires no adjacent Python modules and locates the checkpoint in the parent of its inference folder.
The tested environment uses Python 3.13.15, Torch 2.14.0+cu132, Triton Windows 3.8.0.post28, Transformers 5.17.0, cuDNN 9.24.0 and the separately installed FlashAttention-2 package 2.8.3+cu130torch2.14. See requirements and tested environment.
From the downloaded repository root, install dependencies into your chosen environment:
python -m pip install torch==2.14.0+cu132 --index-url https://download.pytorch.org/whl/cu132
python -m pip install -r .\inference\requirements-windows.txt
Install a Windows FA2 wheel matching your Python, Torch and CUDA environment separately. The tested wheel was flash_attn-2.8.3+cu130torch2.14-cp313-cp313-win_amd64.whl; it is not bundled here. FA2 is the runner's default backend. Its implementation directly calls flash_attn.flash_attn_with_kvcache for both prefill and decode.
Single-turn generation:
python .\inference\mellum_mxfp4.py --prompt "Write a Python function that returns the square of its input." --show-stats
Interactive multi-turn chat:
python .\inference\mellum_mxfp4.py --show-stats
The default configuration is FA2, static 32,768-position BF16 KV, fullgraph TorchInductor default mode and explicit CUDA graph capture/replay. Default output budget is 2,048 tokens. /new resets the conversation, /continue resumes a response paused at the output limit, and /exit quits. Conversation history follows the tokenizer's official chat template and reuses a valid common KV prefix.
Without external FA2, forced cuDNN is another tested option:
python .\inference\mellum_mxfp4.py --attention cudnn --show-stats
For a growing cache with less short-context memory and shorter growth pauses:
python .\inference\mellum_mxfp4.py --kv-cache dynamic --no-compile --graph-mode manual --show-stats
Here --no-compile skips TorchInductor; the Triton kernels still JIT-compile and explicit CUDA graphs remain enabled. Details are in the inference README.
Quantization and execution
- Storage format:
compressed-tensors,mxfp4-pack-quantized. - 5,488 quantized weight matrices; 143 retained BF16/other tensors; 11,119 stored tensors total.
- Two E2M1 values per byte, with the lower nibble representing the even input index.
- One unsigned E8M0 scale byte per 32 values; no NVFP4 global scale is used.
- Architecture: 28 layers, 64 experts per layer with 8 selected per token, 32 query heads and 4 KV heads. Attention combines full attention and a 1,024-token sliding window.
Decode uses packed-weight GEMV on CUDA cores with BF16 activations and FP32 reductions. It is not native FP4 Tensor Core full-model decode. Chunked prefill decodes weight tiles for BF16 Tensor Core matrix multiplication. Weights stay packed in GPU memory; the runtime does not retain a full BF16 copy of the model.
Generic Transformers AutoModelForCausalLM or vLLM loading of this exact checkpoint has not been validated here. The tested execution path is the included custom runner. The quantization manifest records the checkpoint layout.
Local benchmark summary
Native Windows, RTX 5070 Ti Laptop GPU, driver 617.42, batch 1. Contexts were actually prefilled from source text; random or zero KV was not substituted for long prompts. Decode figures are medians of three synchronized timed blocks with greedy autoregressive generation and EOS/BOS suppressed for timing. Interactive chat uses natural EOS.
| Prefilled context tokens | Static KV, compile + graph | Growing KV, compile + graph | Growing KV, eager + graph | Per-token append, eager |
|---|---|---|---|---|
| 128 | 111.03 tok/s | 110.07 tok/s | 110.34 tok/s | 84.11 tok/s |
| 4,096 | 101.32 tok/s | 103.31 tok/s | 100.87 tok/s | 85.53 tok/s |
| 16,384 | 90.27 tok/s | 99.85 tok/s | 91.41 tok/s | 69.23 tok/s |
| 32,000 | 85.19 tok/s | 86.43 tok/s | 87.79 tok/s | 57.47 tok/s |
Static 32K KV occupies 1.75 GiB. The static FA2 runner used 8.68 GiB live Torch GPU allocations, approximately 8.79 GiB reserved. A growing cache used about 7.04 GiB allocated at short context. Torch allocator figures and NVIDIA device-used memory are distinct, especially under Windows WDDM.
Compiled cache-growth events took 7.59–15.23 seconds; eager graph growth took 0.09–0.25 seconds. These pauses are excluded from steady decode rates in the table. Static KV avoids growth events. Prompt processing with static FA2 took 7.19 seconds for 4,096 tokens, 30.62 seconds for 16,384, and 62.95 seconds for 32,000.
With static KV and explicit graphs, short-context default, reduce-overhead and max-autotune measured 111.03, 110.45 and 111.79 tok/s. Native graph modes skipped capture because buffers were mutated. PyTorch also reported insufficient SMs for its GEMM autotuning path on this laptop GPU. Explicit graph capture worked; there was no established speed advantage from max-autotune.
| Decode attention backend | Short-context whole-model tok/s |
|---|---|
| Separately installed FA2 | 111.03 |
| Forced cuDNN SDPA | 87.52 |
| Custom Triton attention | 77.43 |
| xFormers with masked static KV | 26.81 |
| Efficient SDPA | 13.89 |
| Math SDPA | 4.61 |
PyTorch's built-in SDPA FlashAttention backend was unavailable in the tested build; external FA2 worked. FlashInfer CUDA and Tensor Core attention passed isolated probes after configuring MSVC, but were not integrated into a whole-model runner. The custom Triton and xFormers decode configurations used cuDNN for chunked prompt attention.
The laptop reached approximately 78–82 C and clocks varied between runs. Small differences are not a statistically established advantage. This MXFP4 test measured short decode blocks after real prompts; it did not generate one continuous 32,768-token response. Earlier NVFP4 generation measurements are a different workload and are not included in these MXFP4 results.
Validation and limitations
Packed GEMV and fused expert kernels were compared with independently decoded MXFP4 weights. Each of all 28 layers was also checked on three steps with identical hidden inputs and past KV; minimum layer cosine similarity was approximately 0.999995.
The strict cumulative end-to-end reference criterion did not fully pass. Logit cosine similarities on those three steps were 0.999997, 0.989369 and 0.998027, although all three greedy next tokens matched. Per-layer kernel agreement does not establish full-model numerical equivalence or general chat quality. The runtime's validation report records the cumulative criterion separately from its conditioned per-layer acceptance.
Bounded chat regressions passed for interrupted-response continuation, token-stream equality to uninterrupted generation, follow-up answer recall and marker recall after cache growth. These are functional checks, not a broad coding, reasoning or perplexity evaluation.
The source configuration declares a 131,072-position limit. The practical runtime tests here cover approximately 32K context; full 128K operation was not validated. This runner was tested on one native Windows environment and one GPU.
License
Apache 2.0, following the base model. See the base model repository for its model documentation and license.