DeepSeek-V3.1-AWQ-W4AFP8

This model is a W4AFP8 quantized DeepSeek-V3.1 with AWQ quantizaton.
Releated PR：https://github.com/sgl-project/sglang/pull/8573
Releated Project: https://github.com/TMElyralab/sglang/tree/lyra_w4afp8

Benchmark

Test configuration: input/output len = 1000/1000, qps=64, max_concurrency=64, num_prompt=128
Device: H20 * 8
Compared to the original model：

• bs=64，input/output throughput has increased by 56%.
• bs=128，input/output throughput has increased by 125%.

============ Serving Benchmark Result ============
Backend:                                 sglang       
Max request concurrency:                 64        
Successful requests:                     128       
Benchmark duration (s):                  105.50    
Total input tokens:                      128000    
Total generated tokens:                  128000    
Total generated tokens (retokenized):    127551    
Request throughput (req/s):              1.21      
Input token throughput (tok/s):          1213.24   
Output token throughput (tok/s):         1213.24   
Total token throughput (tok/s):          2426.49   
Concurrency:                             63.97     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   52728.31  
Median E2E Latency (ms):                 52728.33  
---------------Time to First Token----------------
Mean TTFT (ms):                          5444.26   
Median TTFT (ms):                        5425.69   
P99 TTFT (ms):                           8768.54   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           47.33     
Median ITL (ms):                         44.18     
P95 ITL (ms):                            46.58     
P99 ITL (ms):                            46.76     
Max ITL (ms):                            7819.3
==================================================

How To Use

Mirror：lmsysorg/sglang:v0.4.6.post5-cu124
or lmsysorg/sglang:v0.5.1.post5-cu126 (cuda12.6 env need to update ptxas to 12.8 on Hopper. reference)

# 1. Repo clone
git clone https://github.com/TMElyralab/sglang && cd sglang
git checkout lyra_w4afp8

# 2. SGLang install
pip install uv cmake
pip install -e "python[all]"

# 3. Recompile sgl-kernel
cd sgl-kernel & make build

# 4. Run SGLang
python3 -m sglang.launch_server --model-path /path/to/DeepSeek-R1-AWQ-W4AFP8 --tp 8 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.9 --quantization w4a8_machete --cuda-graph-max-bs 128 --max-running-requests 128

Citation

We are TMElyralab, the Acceleration Team from Tencent Music Entertainment (TME).

@Misc{TMElyralab_2025,
  author =       {Sa Xiao, Mian Peng, Haoxiong Su, Kangjian Wu, Bin Wu, Yibo Lu, Qiwen Mao, Wenjiang Zhou},
  howpublished = {\url{https://github.com/TMElyralab}},
  year =         {2025}
}

---

DeepSeek-V3.1

---

Introduction

DeepSeek-V3.1 is a hybrid model that supports both thinking mode and non-thinking mode. Compared to the previous version, this upgrade brings improvements in multiple aspects:

• Hybrid thinking mode: One model supports both thinking mode and non-thinking mode by changing the chat template.

• Smarter tool calling: Through post-training optimization, the model's performance in tool usage and agent tasks has significantly improved.

• Higher thinking efficiency: DeepSeek-V3.1-Think achieves comparable answer quality to DeepSeek-R1-0528, while responding more quickly.

DeepSeek-V3.1 is post-trained on the top of DeepSeek-V3.1-Base, which is built upon the original V3 base checkpoint through a two-phase long context extension approach, following the methodology outlined in the original DeepSeek-V3 report. We have expanded our dataset by collecting additional long documents and substantially extending both training phases. The 32K extension phase has been increased 10-fold to 630B tokens, while the 128K extension phase has been extended by 3.3x to 209B tokens. Additionally, DeepSeek-V3.1 is trained using the UE8M0 FP8 scale data format to ensure compatibility with microscaling data formats.

Model Downloads

Chat Template

The details of our chat template is described in tokenizer_config.json and assets/chat_template.jinja. Here is a brief description.

Non-Thinking

First-Turn

Prefix: <｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>

With the given prefix, DeepSeek V3.1 generates responses to queries in non-thinking mode. Unlike DeepSeek V3, it introduces an additional token </think>.

Multi-Turn

Context: <｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>

Prefix: <｜User｜>{query}<｜Assistant｜></think>

By concatenating the context and the prefix, we obtain the correct prompt for the query.

Thinking

First-Turn

Prefix: <｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜><think>

The prefix of thinking mode is similar to DeepSeek-R1.

Multi-Turn

Context: <｜begin▁of▁sentence｜>{system prompt}<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>...<｜User｜>{query}<｜Assistant｜></think>{response}<｜end▁of▁sentence｜>

Prefix: <｜User｜>{query}<｜Assistant｜><think>

The multi-turn template is the same with non-thinking multi-turn chat template. It means the thinking token in the last turn will be dropped but the </think> is retained in every turn of context.

ToolCall

Toolcall is supported in non-thinking mode. The format is:

<｜begin▁of▁sentence｜>{system prompt}{tool_description}<｜User｜>{query}<｜Assistant｜></think> where the tool_description is

## Tools
You have access to the following tools:

### {tool_name1}
Description: {description}

Parameters: {json.dumps(parameters)}

IMPORTANT: ALWAYS adhere to this exact format for tool use:
<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>tool_call_name<｜tool▁sep｜>tool_call_arguments<｜tool▁call▁end｜>{{additional_tool_calls}}<｜tool▁calls▁end｜>

Where:
- `tool_call_name` must be an exact match to one of the available tools
- `tool_call_arguments` must be valid JSON that strictly follows the tool's Parameters Schema
- For multiple tool calls, chain them directly without separators or spaces

Code-Agent

We support various code agent frameworks. Please refer to the above toolcall format to create your own code agents. An example is shown in assets/code_agent_trajectory.html.

Search-Agent

We design a specific format for searching toolcall in thinking mode, to support search agent.

For complex questions that require accessing external or up-to-date information, DeepSeek-V3.1 can leverage a user-provided search tool through a multi-turn tool-calling process.

Please refer to the assets/search_tool_trajectory.html and assets/search_python_tool_trajectory.html for the detailed template.

Evaluation

Note:

• Search agents are evaluated with our internal search framework, which uses a commercial search API + webpage filter + 128K context window. Seach agent results of R1-0528 are evaluated with a pre-defined workflow.

• SWE-bench is evaluated with our internal code agent framework.

• HLE is evaluated with the text-only subset.

Usage Example

import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.1")

messages = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Who are you?"},
    {"role": "assistant", "content": "<think>Hmm</think>I am DeepSeek"},
    {"role": "user", "content": "1+1=?"}
]

tokenizer.apply_chat_template(messages, tokenize=False, thinking=True, add_generation_prompt=True)
# '<｜begin▁of▁sentence｜>You are a helpful assistant<｜User｜>Who are you?<｜Assistant｜></think>I am DeepSeek<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜><think>'

tokenizer.apply_chat_template(messages, tokenize=False, thinking=False, add_generation_prompt=True)
# '<｜begin▁of▁sentence｜>You are a helpful assistant<｜User｜>Who are you?<｜Assistant｜></think>I am DeepSeek<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜></think>'

How to Run Locally

The model structure of DeepSeek-V3.1 is the same as DeepSeek-V3. Please visit DeepSeek-V3 repo for more information about running this model locally.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2024deepseekv3technicalreport,
      title={DeepSeek-V3 Technical Report}, 
      author={DeepSeek-AI},
      year={2024},
      eprint={2412.19437},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.19437}, 
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.