EXL3 quantizations of meta-models/Muse-Glimmer-30B
These quantizations can be used with exllamav3 or vLLM via vllm-exl3-plugin.
They are calibrated broadly, on what models read and write while working (documents and tool output in conversations, code and diffs, the model's own replies) rather than on web text alone.
Bits are allocated per tensor, from the model's measured sensitivity on its own replies, at the same total size as uniform bits.
Actual consumed VRAM may be significantly lower than the total checkpoint size, depending on configuration: see the VRAM columns below.
Which size?
Each bitrate is available as a separate revision of this repository. When planning available VRAM for kv space, refer to the appropriate VRAM column.
The dashed line marks Muse-Glimmer-30B-FP8-block, scored on the same evals as these quants: a familiar point to read them against. It is scored with its weights dequantized under bf16 compute, so it measures the checkpoint's weights, not a deployment (the activation quantization it is served with is not emulated).
| BPW | Size on disk | VRAM (exllamav3) | VRAM (vLLM) | Excess KLD | Δ perplexity |
|---|---|---|---|---|---|
| 2.50 | 14.21 GiB | 11.70 GiB | 8.83 GiB | 0.0792 | +9.56% |
| 3.00 | 15.67 GiB | 13.17 GiB | 10.29 GiB | 0.0386 | +4.38% |
| 3.50 | 17.14 GiB | 14.63 GiB | 11.76 GiB | 0.0188 | +2.25% |
| 4.00 | 18.76 GiB | 16.25 GiB | 13.38 GiB | 0.0095 | +1.00% |
| 5.00 | 21.69 GiB | 19.18 GiB | 16.31 GiB | 0.0025 | +0.24% |
| 6.00 | 24.62 GiB | 22.11 GiB | 19.24 GiB | 0.0005 | +0.04% |
| (base) | 55.46 GiB |
- BPW: Average bits per weight across tensors (not including input embeddings, encoders, MTP draft layers, or output head)
- Size on disk: Total aggregate size of tensors in checkpoint, in GiB
- VRAM (exllamav3): Total VRAM consumption as typically used in exllamav3, in GiB (minus embeddings and MTP draft layers)
- VRAM (vLLM): Total VRAM consumption as typically used in vLLM, in GiB (blockq embeddings, offloaded/omitted encoders, no MTP draft layers)
- Excess KLD: how far each quant's next-token distribution is from the unquantized model's, minus the unquantized model's own rounding noise, so 0 would be indistinguishable from the original (lower is better)
- Δ perplexity: change in perplexity on the same text, same weights (lower is better)
More detail
What each eval slice is, every slice compared on its own, how the composite is built, the calibration, and how to reproduce these quants: TECHNICAL.md.