AxionML Qwen3.5-397B-A17B-NVFP4
Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.
Quantized by NVIDIA. The weights in this repository are an unmodified copy of nvidia/Qwen3.5-397B-A17B-NVFP4-V2 (revision
8f590eae8f10bf55d9a46f79ea0280bde435c9f8). All credit for the quantization belongs to NVIDIA.
This is an NVFP4-quantized version of Qwen/Qwen3.5-397B-A17B (397B total parameters, 17B activated), quantized with NVIDIA Model Optimizer.
About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.
Ready for commercial and non-commercial use under Apache 2.0.
Model Summary
| Architecture | Hybrid-attention MoE (Gated DeltaNet + Gated Attention) with vision encoder |
| Total Parameters | 397B |
| Activated Parameters | 17B |
| Context Length | 262K tokens |
| Checkpoint Size | ~244 GB |
Evaluation Results
| Benchmark | FP8 (baseline) | NVFP4 (V2) |
|---|---|---|
| MMMU Pro | 78.7 | 78.4 |
| GPQA Diamond | 87.2 | 87.7 |
| SciCode | 46.7 | 48.1 |
| AA-LCR | 68.8 | 67.8 |
| IFBench | 76.1 | 76.5 |
| τ²-Bench Telecom | 95.4 | 95.2 |
Scores reported by NVIDIA for this checkpoint. Baseline: Qwen/Qwen3.5-397B-A17B-FP8.
temperature=0.6,top_p=0.95, max 64,000 tokens (128,000 for τ²-Bench Telecom).
Quantization Details
- Quantization format: mixed precision (NVIDIA's V2 recipe) — routed experts in NVFP4 with MSE-based scale search; attention and shared experts in per-tensor FP8
- KV cache: FP8
- Calibration dataset: cnn_dailymail, Nemotron-Post-Training-Dataset-v2
- Tool: NVIDIA Model Optimizer v0.45.0.dev173
- Why V2: NVIDIA's earlier V1 (
nvidia/Qwen3.5-397B-A17B-NVFP4, Feb 2026) quantized only the routed experts with max calibration and left attention and shared experts in BF16 (~251 GB). V2 adds MSE scale search and FP8 attention/shared experts — smaller and faster, and within noise of Qwen's FP8 checkpoint.
Usage
Deploy with SGLang
python3 -m sglang.launch_server \
--model-path AxionML/Qwen3.5-397B-A17B-NVFP4 \
--tensor-parallel-size 4 \
--quantization modelopt_mixed \
--disable-radix-cache \
--trust-remote-code
Validated upstream with lmsysorg/sglang:v0.5.12.post1-cu130 on B300. Tip: add --mamba-ssm-dtype bfloat16 to keep the Mamba SSM state in BF16 for faster decode on Blackwell.
Limitations
The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.
Credits
- Base model: Qwen/Qwen3.5-397B-A17B
- Quantization: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 by NVIDIA
- Mirror: AxionML