Subscribe to my newsletter · Support

Qwen3.8-27B-SLQ-EAR97.8

This is Qwen/Qwen3.8-27B, compressed with a customized version of SLQ (Statistically-Lossless Quantization) and packed for Humming in vLLM. The search combines integer quantization candidates and chooses precision and group size separately for individual projections.

• EAR target: >97.8%; measured packed-model EAR: 97.863974%.
• Weight files: 21.542 GB (21,542,351,288 bytes). This is file size, not total serving memory.
• Mixed integer precisions with BF16 embeddings and activations.
• Original MTP weights retained unchanged in BF16.
• Made by: The Kaitchup.
• License: Apache 2.0.

Method: SLQ paper · SLQ repository. These checkpoints are released by The Kaitchup.

Candidate sources

Candidate weights came from GPTQ, generated locally from Qwen/Qwen3.8-27B, and existing integer refinement variants. Original BF16 weights were also available to the search.

Precision by layer

The map shows actual stored bitwidth and group size. Full-attention Q/K/V share a row. Open the SVG to zoom in.

EAR comparison

EAR means Expected Acceptance Rate. It measures the overlap between the original BF16 model's next-token probabilities and those of the compared model. Here, both distributions are renormalized over the BF16 reference's ten most likely tokens. An EAR of 0.99 means 99% average probability overlap on that measured support; it does not mean 99% benchmark accuracy or identical generated answers.

The plot compares these checkpoints with the previous EAR releases using packed-runtime EAR@10.

SLQ's “statistically lossless” objective concerns approximate fidelity, not identical numerical outputs. These releases meet their stated measured EAR targets; the 97.8% version deliberately uses a looser target. Statistical equivalence on downstream tasks has not been established for these releases.

Instructions: vLLM

Tested for text loading, generation and EAR with vLLM 0.31.0, humming-kernels 0.1.16, and an NVIDIA GH200. Use a vLLM installation with Humming support. The original vision weights and processors are retained; image/video inference and speculative decoding were not included in this validation.

Start an OpenAI-compatible text server:

vllm serve kaitchup/Qwen3.8-27B-SLQ-EAR97.8 \
  --quantization compressed-tensors \
  --linear-backend humming \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --limit-mm-per-prompt '{"image":0,"video":0}'

For reasoning parsing and automatic tool calls, add these server options:

--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Send a chat request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kaitchup/Qwen3.8-27B-SLQ-EAR97.8",
    "messages": [{"role": "user", "content": "Explain expected acceptance rate in one paragraph."}]
  }'

GPU memory must accommodate the weights, runtime buffers and KV cache. The measured scores describe the tested execution path; other runtimes can differ slightly.

Support

If these models are useful, support the work or subscribe to The Kaitchup.