EXL3 quantizations of ornith-ai/Ornith-1.5-9B
These quantizations can be used with exllamav3 or vLLM via vllm-exl3-plugin
Actual consumed VRAM will be lower than the total checkpoint size because of the handling of the input embeddings (and potentially the vision tower). The embeddings are not quantized, because EXL3 is not an efficient format for this purpose. The strategy for this is different for the two platforms:
- For exllamav3, input embeddings are typically offloaded to CPU, which has a performance cost but removes that tensor from VRAM entirely.
- For vllm-exl3-plugin, input embeddings are typically quantized at load time to a format called
blockq, which is a computationally inexpensive and embedding-compatible quantization format. The embeddings are resident in VRAM, but much smaller.
The vision encoder is left unquantized. This preserves full vision fidelity as compared to the base model, which is otherwise harder to characterize. This may represent a significant additional component to VRAM utilization. There are two strategies to eliminate this:
- CPU offload for the encoders. Especially for text-heavy use cases, this represents a minimal performance impact.
- Omitting the encoders entirely, for text-only use cases.
Available bitrates
Each bitrate is available as a separate revision of this repository. When planning available VRAM for kv space, refer to the appropriate VRAM column.
| BPW | Size on disk | VRAM (exllamav3) | VRAM (vLLM) | KLD | PPL |
|---|---|---|---|---|---|
| 2.00 | 5.08 GiB | 3.18 GiB | 2.87 GiB | 0.2906 | 14.0976 |
| 3.00 | 5.88 GiB | 3.99 GiB | 3.67 GiB | 0.0738 | 12.3434 |
| 4.00 | 6.81 GiB | 4.91 GiB | 4.59 GiB | 0.0213 | 12.0635 |
| 5.00 | 7.61 GiB | 5.72 GiB | 5.40 GiB | 0.0062 | 11.9699 |
| 6.00 | 8.41 GiB | 6.52 GiB | 6.20 GiB | 0.0023 | 11.9623 |
| (base) | 17.98 GiB | 0.0009 | 11.9377 |
- BPW: Average bits per weight across tensors (not including input embeddings, encoders, or output head)
- Size on disk: Total aggregate size of tensors in checkpoint, in GiB
- VRAM (exllamav3): Total VRAM consumption as typically used in exllamav3, in GiB (minus embeddings)
- VRAM (vLLM): Total VRAM consumption as typically used in vLLM, in GiB (blockq embeddings, offloaded/omitted encoders)
- KLD: Mean KL divergence from unquantized base model at this bitrate (lower is better)
- PPL: Perplexity at this bitrate (lower is better)
KLD and PPL are quality metrics that represent fidelity to the base model. The values for the base model row are the noise floor: it is not possible for a quantization to meaningfully score better than this value using these metrics.