DeepSeek-R1-0528-GPTQ-4b-128g-experts

Model Overview

This model was obtained by quantizing the weights of deepseek-ai/DeepSeek-R1-0528 to INT4 data type. This optimization reduces the number of bits per parameter from 8 to 4, reducing the disk size and GPU memory requirements by approximately 50%.

Only non-shared experts within transformer blocks are compressed. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization.

Model checkpoint is saved in compressed_tensors format.

Evaluation

This model was evaluated on reasoning tasks (AIME-24, GPQA-Diamond, MATH-500).

Model outputs were generated with the vLLM engine.

For reasoning tasks we estimate pass@1 based on 10 runs with different seeds and temperature=0.6, top_p=0.95 and max_new_tokens=65536.

| | Recovery (%) | deepseek/DeepSeek-R1-0528 | ISTA-DASLab/DeepSeek-R1-0528-GPTQ-4b-128g-experts

(this model) |

| --------------------------- | :----------: | :------------------: | :--------------------------------------------------: |

| AIME 2024

pass@1 | 98.50 | 88.66 | 87.33 |

| MATH-500

pass@1 | 99.88 | 97.52 | 97.40 |

| GPQA Diamond

pass@1 | 101.21 | 79.65 | 80.61 |

| **Reasoning

Average Score | 99.82 | 88.61 | 88.45** |

Contributors

Denis Kuznedelev (Yandex), Eldar Kurtić (Red Hat AI & ISTA), and Dan Alistarh (Red Hat AI & ISTA).