DeepSeek-V3.1-Terminus-W4AFP8

This model is a mixed-precision quantized version of DeepSeek-V3.1-Terminus, with

dense layer keep the FP8 quantization of the original model, while MoE layers

uses INT4 weights and FP8 activation, also called W4AFP8.

Benchmark

The accuracy below was obtained with SGLang V0.5.3 in non-thinking mode.

| Model | math_500 | gpqa | aime2024 | mmlu-pro |

|:------------------------------:|:---------:|:-----:|:---------:|:---------:|

| DeepSeek-V3.1-Terminus-W4AFP8 | 89.83 | 78.28 | 80.0 | 83.66 |

Inference with SGLang

We have already supported deploying this model using tensor parallel in sglang for

better performance. The releated PR

has been merged in SGLang V0.5.2, so you can deploy this model using SGLang

version 0.5.2 or later with tensor parallel.

python3 -m sglang.launch_server --model-path /path/to/DeepSeek-V3.1-Terminus-W4AFP8 --tp 8 --trust-remote-code --host 0.0.0.0 --port 8000