Qwen3.8-27B EXL3 5.0 bpw
EXL3 quantization of Qwen/Qwen3.8-27B (27B parameters) for ExLlamaV3 / TabbyAPI. All credit for the model goes to the Qwen team.
vs the Unsloth GGUFs: lower error, ~1.37x faster
| UD-IQ4_XS GGUF | UD-Q5_K_M GGUF (llama.cpp) | This EXL3 5.0 bpw (ExLlamaV3) | |
|---|---|---|---|
| KL divergence vs BF16, mean | 0.0224 | 0.0069 | 0.0056 (20 % lower than Q5_K_M) |
| KL divergence vs BF16, median | 0.0069 | 0.0014 | 0.0012 |
| KL divergence vs BF16, 99th percentile | 0.232 | 0.061 | 0.045 |
| Same top token as BF16 | 94.0 % | 97.0 % | 97.5 % |
| Decode speed (same GPUs) | 36.5 tok/s | 50.0 tok/s (1.37x) | |
| File size | 14.3 GB (no vision) | 19.8 GB (no vision) | 19 GB (vision included) |
How KL divergence was measured
- Reference: the original BF16 weights, run with ExLlamaV3 (layer streaming).
- Text: WikiText-2 raw test, 20 rows of 2048 tokens. KLD is scored on the second half of each row (20,460 tokens), the same positions llama.cpp uses.
- The BF16 log-probabilities were written in llama.cpp's
--kl-divergence-baseformat, so all quants are compared against the exact same reference and the exact same tokens:- GGUF:
llama-perplexity --kl-divergence-base <bf16 file> --kl-divergence -c 2048 - EXL3: ExLlamaV3
eval/model_diff.py(A = EXL3, B = BF16), restricted to the same positions.
- GGUF:
- "Same top token" for the EXL3 is computed over all positions of the rows (llama.cpp's figure uses the second half only), so treat that line as indicative.
How speed was measured
RTX 5060 Ti 16 GB + RTX 3070 8 GB, both models split across the two GPUs, MTP speculative decoding (2 draft tokens for the GGUF, 3 for the EXL3), temperature 1, reasoning on. Same script for both: 4 prompts (3 coding, 1 explanation), 600 tokens each, run twice, average decode speed after the first token.
| Runs | |
|---|---|
| GGUF UD-Q5_K_M | 34.7, 33.7, 43.5, 32.0, 35.7, 35.4, 40.7, 36.1 |
| EXL3 5.0 bpw | 60.2, 42.8, 50.2, 40.9, 63.6, 41.0, 61.3, 39.8 |
The EXL3 run used a tensor-parallel split tuned for this GPU pair (41/59, attention on the 16 GB card) and a 64k context with an 8-bit cache. With a default split, expect somewhat less.
Quantization
| Part | Bits |
|---|---|
| Decoder layers | 5.0 bpw |
| Output head | 6 bpw |
| Vision tower | 6 bpw |
| MTP head | 5 bpw |
| Embeddings | 16 bit |
Settings: -b 5.0 -hb 6 -vb 6 -mb 5 --out_scales always, default calibration. Size: ~19 GB.
The MTP head and the vision tower are included.
Fitting in 24 GB
With 24 GB of VRAM in total (16 + 8 GB here), the weights take ~19 GB, which leaves room for about 64k tokens of context with an 8-bit cache. A 131k context did not fit on this setup.
License
Same as the base model: Apache 2.0.