DeepSeek-V3-0324-GPTQ-4b-128g-experts
Model Overview
This model was obtained by quantizing the weights of deepseek-ai/DeepSeek-V3-0324 to INT4 data type. This optimization reduces the number of bits per parameter from 8 to 4, reducing the disk size and GPU memory requirements by approximately 50%.
Only non-shared experts within transformer blocks are compressed. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization.
Model checkpoint is saved in compressed_tensors format.
| Models | Experts Quantized | Attention blocks quantized | Size (GB) |
|---|---|---|---|
| deepseek-ai/DeepSeek-V3-0324 | ❌ | ❌ | 671 GB |
| ISTA-DASLab/DeepSeek-V3-0324-GPTQ-4b-128g-experts | ✅ | ❌ | 346 GB |
Contributors
Denis Kuznedelev (Yandex), Eldar Kurtić (Red Hat AI & ISTA), Jiale Chen (ISTA), Michael Goin (Red Hat AI), Elias Frantar (ISTA), Dan Alistarh (Red Hat AI & ISTA).