AxionML DeepSeek-V4-Pro-0813-NVFP4
Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.
Quantized by NVIDIA. The weights in this repository are an unmodified copy of nvidia/DeepSeek-V4-Pro-0813-NVFP4 (revision
949138637e8e8fe190335be2396af62187dc8a83). All credit for the quantization belongs to NVIDIA.
This is an NVFP4-quantized version of deepseek-ai/DeepSeek-V4-Pro-0813 (1.65T total parameters, 49B activated), quantized with NVIDIA Model Optimizer.
About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.
Ready for commercial and non-commercial use under MIT.
Model Summary
| Architecture | MoE with hybrid attention (Compressed Sparse + Heavily Compressed Attention), Manifold-Constrained Hyper-Connections |
| Total Parameters | 1.65T |
| Activated Parameters | 49B |
| Experts | 384 routed |
| Speculative Decoding | DSpark heads included (unquantized) |
| Reasoning Effort | low / high / max |
| Context Length | 1M tokens |
| Checkpoint Size | ~941 GB |
Evaluation Results
| Benchmark | MXFP4 (source) | NVFP4 |
|---|---|---|
| GPQA Diamond | 88.51 | 88.42 |
| AA-LCR | 68.67 | 69.33 |
| τ²-Bench Telecom | 96.49 | 98.25 |
| SciCode | 53.45 | 53.75 |
| IFBench | 76.53 | 75.68 |
| Terminal-Bench Hard | 51.39 | 50.69 |
Scores reported by NVIDIA for this checkpoint (SGLang, B200). Baseline: deepseek-ai/DeepSeek-V4-Pro-0813.
temperature=1.0,top_p=1.0, max reasoning effort.
Quantization Details
- Quantization format: routed MoE experts in NVFP4 (weights and activations); source MXFP4 expert weights are bit-cast losslessly to NVFP4 and only the block scales are rewritten; FP8 attention and shared experts, BF16 norms and embeddings unchanged
- Calibration dataset: 1,024 samples (sequence cap 4,096) from cnn_dailymail and Nemotron-Post-Training-Dataset-v2
- Tool: NVIDIA Model Optimizer v0.47.0rc1 (recipe)
Usage
Deploy with SGLang
python3 -m sglang.launch_server \
--model-path AxionML/DeepSeek-V4-Pro-0813-NVFP4 \
--tp 8 \
--trust-remote-code \
--tool-call-parser deepseekv4 \
--reasoning-parser deepseek-v4
Deploy with vLLM
vllm serve AxionML/DeepSeek-V4-Pro-0813-NVFP4 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--reasoning-parser deepseek_v4 \
--max-model-len 400000 \
--gpu-memory-utilization 0.9 \
--kv-cache-dtype fp8 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill
vLLM configuration validated upstream on 8x B200 with vllm/vllm-openai:v0.27.1 and v0.28.0. DSpark heads are preserved, but speculative decoding was not validated for this NVFP4 checkpoint.
Limitations
The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.
Credits
- Base model: deepseek-ai/DeepSeek-V4-Pro-0813
- Quantization: nvidia/DeepSeek-V4-Pro-0813-NVFP4 by NVIDIA
- Mirror: AxionML