Qwen3.6-27B-W4A16-G128

W4A16 GPTQ quantization of Qwen/Qwen3.6-27B, targeted at single-GPU vLLM serving on Ada-class GPUs (L40S, RTX 6000 Ada, RTX 4090) where the FP8 checkpoint leaves too little VRAM for usable KV cache at long context.

TL;DR

Quantization wins on every measured axis except single-stream prefill TTFT (+13 %, gptq_marlin dequant cost during prefill - mitigated under concurrent load).

What is quantized

Quantized (W4A16, group_size=128, symmetric, desc_act on):

• All model.language_model.layers.{0..63}.self_attn.{q,k,v,o}_proj (16 full-attention layers)
• All model.language_model.layers.{0..63}.linear_attn.{in_proj_qkv,in_proj_z,out_proj} (48 Gated-DeltaNet layers)
• All model.language_model.layers.{0..63}.mlp.{gate,up,down}_proj

Kept BF16:

• Vision tower (model.visual.* - all 27 SigLIP-style blocks + merger)
• MTP head (mtp.* - 15 tensors, 0.85 GiB) - required for speculative decoding correctness
• lm_head.weight and model.language_model.embed_tokens.weight
• All *_norm and *.conv1d.weight (Mamba conv)

Quantization recipe

• Tool: gptqmodel 6.0.3
• Stack: transformers 5.6.0, torch 2.11.0+cu130, torchvision 0.26.0
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k, max_seq_length=2048, applied with the model's chat template.
• Hyperparameters: bits=4, group_size=128, sym=True, desc_act=True, damp_percent=0.01
• Hardware: single L40S 46 GiB (sm_89, Ada Lovelace), driver 595.58.03, CUDA 13.2.
• Wall clock: 53.4 minutes.
• Per-layer GPTQ losses: 1e-9 to 1e-5 (very small - clean fit on hybrid attention).

vLLM serving

Tested on vllm 0.19.1 with transformers 5.6.0+. The compressed-tensors config is auto-detected; no --quantization flag required.

vllm serve LibertAIDAI/Qwen3.6-27B-W4A16-G128 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 8 \
  --max-model-len 200000 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --limit-mm-per-prompt '{"image":4,"video":0,"audio":0}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

Drop the --speculative-config flag if you do not want MTP - saves ~0.5 GiB VRAM (~4K extra KV tokens) at the cost of ~2x slower single-stream decode.

Caveats

• Calibration was English+chat-only (ultrachat_200k). Multilingual and code-heavy workloads have not been measured against a held-out set; expect quality on par with other community W4A16 GPTQ Qwen quants but not specifically guaranteed.
• Vision left BF16 - vision-only quality should match upstream Qwen3.6-27B-FP8 closely. Not redundantly evaluated.
• MTP mtp.safetensors is shipped as a separate file referenced from model.safetensors.index.json. Keep both alongside the main shards or vLLM will boot without spec decoding (silent - no error, just 2x slower decode).
• Tested only on Ada (L40S, sm_89). Should also work on Hopper, Blackwell, and Ampere via gptq_marlin, but not benchmarked there.
• desc_act=True may slightly increase load time vs desc_act=False; chosen for the small accuracy benefit on long-tail distributions in linear-attention layers.

Files

• model-{1..5}-of-00005.safetensors - main quantized weights (~18 GiB)
• mtp.safetensors - BF16 MTP draft head (0.85 GiB), referenced from model.safetensors.index.json
• quantize_config.json - gptqmodel config
• quant_log.csv - per-module quantization losses + timings
• Standard tokenizer / preprocessor / chat-template files inherited from upstream

License & attribution

Inherits Apache 2.0 from Qwen/Qwen3.6-27B. All rights, responsibilities, and acceptable-use policies of the upstream license apply.

Quantization performed by LibertAI. This artifact is in production use serving the qwen3.6-27b model alias on the LibertAI inference platform.