Qwen3.8-27B — EXL3 5.5 bpw (AWQ-smoothed)

Trellis quantization (EXL3, a QTIP variant) of Qwen/Qwen3.8-27B at 5.5 bits per weight, with an AWQ smoothing pre-pass applied to the BF16 weights first.

20.0 GB, lm_head at 6 bpw, MTP layers quantized inline.

Requires exllamav3 / TabbyAPI. EXL3 cannot be served by vLLM.

Measured quality

Every row measured identically: 231 documents, 142,727 scored token positions, against the BF16 base. this model rows are the three checkpoints in this series; the others are for context.

* EXL3 throughput is single-stream argmax on one GPU via exllamav3 — not comparable to the batched vLLM figures in the other rows. It was not tuned.

How to read this

Columns are disagreement rates against the BF16 base, split by how confident the base model was at that position (its top1−top2 logprob margin):

Only confident and certain matter. divmed is the median token index at which free greedy generation first diverges from the base model (higher = holds a trajectory longer).

Bitrate scaling in this series

5.5 bpw is the knee of the curve. Above it, extra bits buy an order of magnitude less per bit.

Recipe

1. AWQ smoothing on the BF16 weights — a pure, function-preserving weight transform (fold 1/s into the preceding norm, s into the next linear's input dim), calibrated on 256 sequences. Verified function-preserving: the ratio to the original weights is a separable per-row × per-column scale.
2. EXL3 conversion at 5.5 bpw on the smoothed weights.

AWQ stacks because it leaves an ordinary BF16 checkpoint. GPTQ and AutoRound do not stack — they are quantizers whose error compensation is tied to specific rounding decisions on a specific grid, which EXL3 discards while keeping the damage.

Honest comparison

• Beats every NVFP4 build measured, at smaller size (5.5 bpw: 1.34% at 20.0 GB vs NVFP4's 1.85% at 24.7 GB, z=+7.11).
• 6.5 bpw takes the best top-1, near-tie, moderate and divmed in the field — it agrees with the base model more often overall.
• But INT4 group-32 asymmetric is still more faithful where it counts: 0.93% confident vs 1.23% (z=−5.06), and 0.09% vs 0.15% certain (z=−3.18). That build serves in vLLM at 4,617 tok/s.

Significance is paired McNemar over matched token positions, n = 33,812.

KV-cache quantization (calibrated scales included)

This checkpoint ships per-layer k_scale / v_scale in model-kv-scales.safetensors, with kv_cache_scheme recorded under quantization_config — the on-disk contract llm-compressor emits for kv_cache_scheme and vLLM reads in BaseKVCacheMethod. Without them vLLM falls back to a scale of 1.0.

--kv-cache-dtype fp8      # 32 KiB/token, from 64 KiB at fp16
--kv-cache-dtype nvfp4    # 18 KiB/token, FlashInfer + TRTLLM attention only

Qwen3.8-27B is a hybrid stack: only 16 of its 64 layers are full_attention and carry a KV cache at all, so this is 32 scalars in a 4 KB file.

Scales were observed on this quantized model, post-RoPE, where vLLM caches K/V, over a 2,353-document / 3,711,425-token corpus, taking a running absmax per layer (memoryless_minmax, llm-compressor's default KV observer); k_scale = amax/448.

Corpus size turned out to matter, and only for V. absmax is a tail statistic — it sits flat and then jumps when a rare document appears, so a small calibration set looks converged when it is not. A 48-document pass under-estimated V by up to 1.9x, and a 231-document pass was still up to 1.5x low; at this size no tensor set a new maximum over the final 50 documents. Under-estimating clips exactly the outliers a cache most needs to keep. K is converged within ~100 documents and needs none of this; V's range also grows ~13x from the first attention layer to the last, which is why the scales are per-layer.

Measured cost

Disagreement vs the BF16 base bucketed by the base model's top1−top2 margin, same corpus and harness as the rest of the matrix. confident is the pre-registered decider; z is paired McNemar against the fp16-cache row, n = 33,812, positive = fp16 cache better.

Note this harness scores next-token agreement, which is not sensitive to what a clipped KV outlier does to a long reasoning chain. It supports "fp8 KV is cheap here"; it is not a substitute for a reasoning benchmark.