Mellum2.1-12B-A2.5B-Thinking — NVFP4
Weight-only NVFP4 quantization of JetBrains/Mellum2.1-12B-A2.5B-Thinking, with a tested native Windows, single-stream inference runtime.
The checkpoint is stored in the compressed-tensors NVFP4 format as one model.safetensors file: 6.676 GiB / 7,168,153,484 bytes. The included runtime keeps weights packed, uses BF16 activations and KV, and supports persistent multi-turn chat with static cache allocation, FlashAttention-2 or cuDNN/SDPA, Triton kernels, torch.compile, and CUDA graphs.
Measured on one RTX 5070 Ti Laptop GPU: approximately 118 tokens/sec in short multi-turn chat and 84 tokens/sec at the end of a continuous 32,768-token generation test. These are local runtime measurements, not model-quality scores or performance guarantees for other systems.
Quick start
The tested path is the included single-file Python runtime. Generic Transformers AutoModelForCausalLM or vLLM loading of this exact checkpoint has not been validated here.
Download the repository locally, then use a dedicated environment. From the downloaded repository root:
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install torch==2.14.0+cu132 --index-url https://download.pytorch.org/whl/cu132
.\.venv\Scripts\python.exe -m pip install -r requirements-windows.txt
Start chat with the cuDNN backend, which does not require the separate FlashAttention package:
.\.venv\Scripts\python.exe inference\mellum_single.py --model . --attention cudnn
For the faster measured FA2 configuration, install a Windows FlashAttention wheel matching your environment, then select --attention fa2. The tested binary was flash_attn-2.8.3+cu130torch2.14-cp313-cp313-win_amd64.whl; it is not bundled here. A generic pip source installation is not a verified reproduction of that Windows build.
.\.venv\Scripts\python.exe inference\mellum_single.py --model . --attention fa2
Type messages at You:. /new starts a fresh conversation without reloading the model; /exit quits. Responses stream and stop normally at EOS. The default output limit is 512 tokens including reasoning; use --max-new-tokens 1024 for longer responses. Prior turns and generated reasoning remain in KV, and follow-ups process only their new prompt tokens.
See installation, runtime options and benchmark reproduction and the exact tested environment.
Checkpoint format
| Item | Representation |
|---|---|
| Packed weights | FP4 E2M1, two elements per uint8 |
| Block size | 16 elements |
| Block scales | FP8 E4M3 |
| Global scales | FP32 |
| Activations in the included runtime | BF16 |
| KV in the included runtime | BF16 |
| Container | Single model.safetensors |
| Quantization metadata | compressed-tensors, nvfp4-pack-quantized, version 0.19.0 |
Quantized matrices are the MoE expert gate/up/down projections, attention Q/K/V/output projections, and LM head. Embeddings, router weights, and normalization weights remain BF16.
The complete checkpoint has 5,489 NVFP4 matrices, 142 BF16 tensors, and 16,609 stored tensor objects. Each quantized matrix uses:
weight_packed # packed FP4 values
weight_scale # FP8 block scales
weight_global_scale # FP32 global encoding scale
In this checkpoint, even input elements occupy the low nibble. Reconstruction multiplies the FP4 values by their block scales and divides by the stored global encoding scale. Backend scale swizzling and global-scale conventions must be handled explicitly. The included runtime preserves the independent gate/up scales rather than substituting one fused global scale.
The configuration declares an nvfp4_expanded_mse weight observer. Quantization/calibration scripts and a calibration dataset are not provided in this repository; the evidence below validates the artifact and runtime, rather than claiming full quantization reproducibility.
Model architecture
The underlying model is a 12B mixture of experts with approximately 2.5B active parameters per token. It has 28 layers, 64 experts, eight selected experts per token, hidden size 2304, expert intermediate size 896, 32 query heads and four KV heads. Twenty-one layers use 1,024-token sliding attention; seven use full attention.
The upstream model advertises 131,072 positions. The included runtime was tested with 32K chat capacity and a slightly larger cache for a 32K-new-token stress test. Full 131K operation on the tested 12 GB GPU is not validated. For original training, intended uses, evaluations and limitations, see the upstream model card.
Runtime implementation
The runtime is one Python file and runs natively on Windows, without WSL. Decode uses packed-weight CUDA-core GEMV kernels: weight tiles are unpacked/scaled in registers and multiplied by BF16 activations with FP32 reductions. Chunked prefill uses BF16 Tensor Core matmuls. The runtime does not claim native FP4 Tensor Core GEMM for the full-model decode path and does not keep a resident BF16 copy of the complete model.
GPU routing selects eight experts, fused gate/up kernels retain each projection's scale, and grouped down kernels reduce the expert outputs. Q/K head normalization, both RoPE modes, GQA and sliding-window masking are retained. The default chat allocates 32,768 BF16 KV positions in every layer and reuses the cache across turns.
The decode function is compiled with torch.compile(fullgraph=True) and captured in an explicit CUDA graph. Prefill processes 128 tokens from the one sequence at a time; this is still batch 1. The CLI currently uses greedy decoding and does not provide an HTTP server, sampling controls, or multi-request serving.
Tested environment
| Component | Tested version |
|---|---|
| GPU | RTX 5070 Ti Laptop, SM120, 12 GB class |
| OS / Python | Native Windows x64 / Python 3.13.15 |
| Driver | 617.14 |
| PyTorch / CUDA runtime | 2.14.0+cu132 / 13.2 |
| cuDNN | 9.24.0 |
| Triton Windows | 3.8.0.post28 |
| FlashAttention | 2.8.3+cu130torch2.14 |
| Transformers / Safetensors | 5.17.0 / 0.8.0 |
Other versions, operating systems, GPUs and binary combinations have not been validated. The default 32K FA2 chat test peaked at approximately 8.70 GiB allocated / 8.88 GiB reserved in PyTorch. CUDA/library/display allocations add to the PyTorch figures.
Performance measurements
Persistent chat and actual long input
| Workload, batch 1 | Measurement |
|---|---|
| FA2 short multi-turn chat, median decode | 118.48 tokens/sec |
| FA2 response after processing 30,060 source-code prompt tokens | 89.39 tokens/sec |
| FA2 follow-up with 30,231 tokens already cached | 90.72 tokens/sec |
| Follow-up prompt / time to first prediction | 22 new tokens / 0.103 seconds |
| Initial 30,060-token source-code prefill | 51.31 seconds, approximately 586 tokens/sec |
These were real processed prompts and generated responses. No fabricated KV tensors were inserted. Initial loading and compilation are outside the decode timing. Generation counts include reasoning and special tokens. A long initial prompt still has substantial first-token latency; subsequent turns reuse its KV.
Continuous 32,768-token generation
The sweep used a 58-token prompt, real greedy autoregressive generation, and a static capacity of 32,896 positions per layer to fit prompt plus output. EOS was suppressed for the requested minimum length; ordinary chat retains normal EOS stopping. The stream contained exactly 32,768 emitted tokens and zero EOS tokens. Its tail became repetitive, so this is a throughput/context stress test rather than a long-answer quality demonstration.
| Generated-token point | Recent decode tokens/sec | ms/token |
|---|---|---|
| 0 — first timing window | 120.01 | 8.33 |
| 1,024 | 109.08 | 9.17 |
| 2,048 | 101.40 | 9.86 |
| 4,096 | 94.55 | 10.58 |
| 8,192 | 91.28 | 10.96 |
| 16,384 | 89.22 | 11.21 |
| 32,768 | 84.16 | 11.88 |
The 32,767 timed post-prefill decode steps took 366.70 seconds, averaging 89.36 tokens/sec. Prefill predicts the first emitted token. Point 0 is the initial 256-step window; other checkpoints use the trailing window, normally 256 steps. Measurements were saved every 1,024 tokens. Output was buffered during the sweep.
At the checkpoints, PyTorch memory stayed at 8.696 GiB allocated / 8.998 GiB reserved. Peaks were 8.702 / 9.000 GiB. Peak sampled device usage was 9,777 MiB, approximately 9.55 GiB, and maximum sampled GPU temperature was 82°C. The curve includes both context growth and sustained laptop clock changes; it is not an isolated attention-scaling experiment.
See CSV checkpoints, complete timing/telemetry JSON, token IDs, generated text, and benchmark methodology.
Validation and quality limits
- Checkpoint tensor names/counts and packed dtype coverage were audited.
- Individual packed projections were compared with reconstructed-weight Torch calculations, including changing GPU expert IDs.
- Three early full-model token steps were compared with independently reconstructed NVFP4-to-BF16 Torch linears and SDPA. Logit cosines were 0.999996, 0.999484 and 0.999944; all three next-token choices matched.
- Q/K normalization, RoPE and cache writes matched the tested reference cases. Sliding-cache rollover, grouped prompt routing, and attention were checked.
- A cached 402-token conversation followed by 24 new tokens matched a fresh rebuild with logit cosine 0.999985 and the same next-token choice.
- Multi-turn recall, streamed output, EOS stopping and conversation reset were tested.
These checks establish representation and implementation compatibility for the tested cases. They are not a broad evaluation of the quantized model's coding, reasoning, safety or long-context quality. Upstream benchmark scores have not been rerun on this quantization and should not be presented as its measured scores. See the evidence files.
Repository files
The model, configuration, tokenizer and original chat-template files are at the root. inference/ contains the single-file runtime, setup guide and environment record. benchmarks/ contains measured results and validation evidence. The original large weight file is unchanged by the runtime/documentation update.
License and acknowledgements
Apache License 2.0, following the upstream model. See LICENSE and NOTICE.md. JetBrains developed and trained the original model. This repository provides its NVFP4 derivative and the local inference runtime; dependency packages retain their own licenses.