Melvin56/DeepScaleR-1.5B-Preview-GGUF
Original Model : agentica-org/DeepScaleR-1.5B-Preview
All quants are made using the imatrix option.
| CPU (AVX2) | Metal | cuBLAS | rocBLAS | SYCL | CLBlast | Vulkan | Kompute | |
|---|---|---|---|---|---|---|---|---|
| K-quants | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ 🐢5 | ✅ 🐢5 | ❌ |
| I-quants | ✅ 🐢4 | ✅ 🐢4 | ✅ | ✅ | Partial¹ | ❌ | ❌ | ❌ |
✅: feature works.
🚫: feature does not work
❓: unknown, please contribute if you can test it youself
🐢: feature is slow
¹: IQ3_S and IQ1_S, see #5886
²: Only with -ngl 0
³: Inference is 50% slower
⁴: Slower than K-quants of comparable size
⁵: Slower than cuBLAS/rocBLAS on similar cards
⁶: Only q8_0 and iq4_nl
| Model | Size (GB) |
|---|---|
| Q2_K | 0.75 |
| IQ3_XXS | 0.76 |
| IQ3_XS | 0.83 |
| IQ3_S | 0.86 |
| IQ3_M | 0.87 |
| Q3_K_M | 0.92 |
| IQ4_XS | 1.01 |
| Q4_K_M | 1.12 |
| Q5_K_M | 1.28 |
| Q6_K | 1.46 |
| Q8_0 | 1.89 |
| F16 | 3.55 |