Qwopus3.8-27B-Coder-Flash EXL3 4.0 bpw

EXL3 quantization of Jackrong/Qwopus3.8-27B-Coder-Flash (27B parameters) for ExLlamaV3 / TabbyAPI. All credit for the model goes to Jackrong.

vs the official Q4_K_M GGUF: half the error, ~1.5x faster

	Official Q4_K_M GGUF (llama.cpp)	This EXL3 4.0 bpw (ExLlamaV3)
KL divergence vs BF16, mean	0.0229	0.0116 (2x lower)
KL divergence vs BF16, median	0.0085	0.0037
KL divergence vs BF16, 99th percentile	0.200	0.100
Same top token as BF16	93.8 %	95.6 %
Decode speed (same GPUs)	37.6 tok/s	58.5 tok/s (1.56x)
File size	16.8 GB (no vision)	16 GB (vision included)

How KL divergence was measured

Reference: the original BF16 weights, run with ExLlamaV3 (layer streaming).
Text: WikiText-2 raw test, 40 rows of 2048 tokens. KLD is scored on the second half of each row (40,920 tokens), the same positions llama.cpp uses.
The BF16 log-probabilities were written in llama.cpp's --kl-divergence-base format, so both quants are compared against the exact same reference and the exact same tokens:
- GGUF: llama-perplexity --kl-divergence-base <bf16 file> --kl-divergence -c 2048
- EXL3: ExLlamaV3 eval/model_diff.py (A = EXL3, B = BF16), restricted to the same positions.
"Same top token" for the EXL3 is computed over all positions of the rows (llama.cpp's figure uses the second half only), so treat that line as indicative.

How speed was measured

RTX 5060 Ti 16 GB + RTX 3070 8 GB, both models split across the two GPUs, MTP speculative decoding with 2 draft tokens for both, temperature 1, reasoning on. Same script for both: 4 prompts (3 coding, 1 explanation), 600 tokens each, run twice, average decode speed after the first token.

	Runs
GGUF Q4_K_M	41.0, 33.7, 43.0, 38.0, 38.7, 34.7, 35.6, 36.2
EXL3 4.0 bpw	63.6, 62.9, 66.8, 51.0, 57.0, 57.6, 54.2, 54.7

The EXL3 run used a tensor-parallel split tuned for this GPU pair (41/59, attention on the 16 GB card). With a default split, expect somewhat less.

Quantization

Part	Bits
Decoder layers	4.0 bpw
Output head	5 bpw
Vision tower	6 bpw
MTP head	4 bpw
Embeddings	16 bit

Settings: -b 4.0 -hb 5 -vb 6 -mb 4 --out_scales always, mul1 codebook, default calibration (250 rows x 2048). Size: ~16 GB.

The MTP head and the vision tower are included.

MTP draft tokens

Draft tokens	Decode
1	~50 tok/s
2	~59 tok/s
3	~54 tok/s

With this fine-tune, 2 draft tokens works best: the MTP head accepts fewer tokens than on the original Qwen3.8 27B, so longer drafts waste work.

License

Same as the base model: Apache 2.0.

nathaninline/Qwopus3.8-27B-Coder-Flash-exl3-4.0bpw