Subscribe to my newsletter · Support

Qwen3.8-27B-SLQ-EAR99

This is Qwen/Qwen3.8-27B, compressed with a customized version of SLQ (Statistically-Lossless Quantization) and packed for Humming in vLLM. The search combines several quantization sources and chooses precision separately for individual projections, including selected attention heads.

• EAR target: 99%; measured packed-model EAR: 99.002514%.
• Weight files: 30.518 GB (30,518,225,504 bytes). This is file size, not total serving memory.
• Mixed integer precisions with selected original BF16 weights; BF16 activations.
• Original MTP weights retained unchanged in BF16.
• Made by: The Kaitchup.
• License: Apache 2.0.

Method: SLQ paper · SLQ repository.

Candidate sources

Candidate weights came from GPTQ (INT2–INT8), generated locally from Qwen/Qwen3.8-27B; kaitchup/Qwen3.8-27B-hum-high-MTP-17.7GB, quantized with AutoRound; and ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ. Original BF16 weights were also available to the search.

Precision by layer

The map shows actual stored bitwidth. Open the SVG to zoom in.

EAR comparison

EAR means Expected Acceptance Rate. It measures the overlap between the original BF16 model's next-token probabilities and those of the compared model. Here, both distributions are renormalized over the BF16 reference's ten most likely tokens. An EAR of 0.99 means 99% average probability overlap on that measured support; it does not mean 99% benchmark accuracy or identical generated answers.

All measured rows use the same reference and metric. The execution column distinguishes packed/native inference from measurements of dequantized weights.

SLQ's “statistically lossless” objective concerns approximate fidelity, not identical numerical outputs. These releases meet their stated measured EAR targets; the 98% version deliberately uses a looser target. Statistical equivalence on downstream tasks has not been established for these releases.

Instructions: vLLM

Tested for text loading, generation and EAR with vLLM 0.31.0, humming-kernels 0.1.16, and an RTX PRO 6000 Blackwell. Use a vLLM installation with Humming support. The original vision weights and processors are retained; image/video inference and speculative decoding were not included in this validation.

Start an OpenAI-compatible text server:

vllm serve kaitchup/Qwen3.8-27B-SLQ-EAR99 \
  --quantization humming \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --limit-mm-per-prompt '{"image":0,"video":0}'

For reasoning parsing and automatic tool calls, add these server options:

--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Send a chat request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kaitchup/Qwen3.8-27B-SLQ-EAR99",
    "messages": [{"role": "user", "content": "Explain expected acceptance rate in one paragraph."}]
  }'

GPU memory must accommodate the weights, runtime buffers and KV cache. The measured scores describe the tested execution path; other runtimes can differ slightly.

Support

If these models are useful, support the work or subscribe to The Kaitchup.