EXL3 quantizations of Qwen/Qwen3.8-27B

These quantizations can be used with exllamav3 or vLLM via vllm-exl3-plugin.

They are calibrated broadly, on what models read and write while working (documents and tool output in conversations, code and diffs, the model's own replies) rather than on web text alone, and measured with agentic and coding work as the primary use case.

Bits are allocated per tensor, from the model's measured sensitivity on its own replies, at the same total size as uniform bits.

Earlier revisions of this repository (uniform bits, the same calibration) are superseded: at equal size the current ones cut composite excess KLD by 4-12%. They remain in each branch's history.

Actual consumed VRAM may be significantly lower than the total checkpoint size, depending on configuration: see the VRAM columns below.

---

Which size?

Each bitrate is available as a separate revision of this repository. When planning available VRAM for kv space, refer to the appropriate VRAM column.

The dashed line marks Qwen3.8-27B-FP8, scored on the same evals as these quants: a familiar point to read them against. It is scored with its weights dequantized under bf16 compute, so it measures the checkpoint's weights, not a deployment (the activation quantization it is served with is not emulated).

• BPW: Average bits per weight across tensors (not including input embeddings, encoders, MTP draft layers, or output head)
• Size on disk: Total aggregate size of tensors in checkpoint, in GiB
• VRAM (exllamav3): Total VRAM consumption as typically used in exllamav3, in GiB (minus embeddings and MTP draft layers)
• VRAM (vLLM): Total VRAM consumption as typically used in vLLM, in GiB (blockq embeddings, offloaded/omitted encoders, no MTP draft layers)
• Excess KLD: how far each quant's next-token distribution is from the unquantized model's, minus the unquantized model's own rounding noise, so 0 would be indistinguishable from the original (lower is better)
• Δ perplexity: change in perplexity on the same text, same weights (lower is better)

Against other quants of this model

This section appears because other quantizations of this model exist (turboderp/Qwen3.8-27B-exl3); it is the case for publishing another set.

Every set is plotted relative to the same baseline: an uncalibrated EXL3 quantization of this model at each bit rate, which favors no calibration. Each set keeps its own draw-to-draw noise (about 1-2% here) rather than one being drawn as a perfect line, and the baseline's own noise moves all of them together, so their order at any size is unaffected.

Ratios of excess KLD, below 1.00 better: the composite ratio is the geometric mean of the two composite slices' ratios at their weights, so each slice counts as weighted whatever its absolute scale. Each revision is compared with the most competitive quant at the same nominal bit rate among the sets above; the size column shows how much larger (+) or smaller (-) this card's checkpoint is than that one. The worst eval slice is the largest ratio over every eval slice: real user prompts (WildChat), real agent sessions (Open-SWE-Traces), documents in a user turn, documents as tool results, multilingual documents, tool-use loops, the model's own voice. At or below 1.00, no slice is worse. The slices beyond the two in the composite are constructed conversations, with documents and tool definitions held out of calibration; they locate an effect rather than measure real use.

* No other quant at this bit rate: compared against the most competitive set's ladder, interpolated (log scale) at equal size, counting the transformer body only, since output-head bits can differ between rungs.

† Within about twice the unquantized model's own rounding noise: both quants here are, in practice, indistinguishable from the original, and the ratio compares differences too small to matter in use. Read it as noise, not as a ranking.

‡ A composite slice is left out where both quants sit within about twice the unquantized model's own rounding noise on it; the ratio is over the other slice alone: 5.00 bpw (real agent sessions (Open-SWE-Traces) left out).

More detail

What each eval slice is, every slice compared on its own, how the composite is built, the calibration, and how to reproduce these quants: TECHNICAL.md.