Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency

An INT4 (W4A16) build of Qwen/Qwen3.6-35B-A3B made for spoken conversation — voice agents where every turn has to come back in well under half a second. Everything a decode step actually reads (routed and shared experts, Gated-DeltaNet and attention projections) is INT4, lm_head is INT8, the calibration data is spoken Chinese dialogue with thinking off, and the MTP head is kept in BF16.

TL;DR — 21.5 GB on disk, 18.6 GiB of weights in vLLM (the popular experts-only INT4 checkpoints: 24.5–25.5 GB / 22.2 GiB) · compressed-tensors INT4 sym g128 → vLLM Marlin MoE kernel on SM 8.0+ GPUs (Hopper: see the quick-start notes), zero-config (measured on vLLM 0.24.0, 0.27.1, 0.30.0, 0.31.0 and 0.32.0) · one RTX 5090, single stream, ~1.5 k-token chat prompt, 20–30-token replies: TTFT 53 ms, 3.3 ms/token vs 4.6 ms/token for the experts-only, data-free QuantTrio/Qwen3.6-35B-A3B-AWQ build: −29 % decode, same TTFT; 5 concurrent streams 9.4 vs 12.2 ms/token. On public benchmarks it is level, within sampling noise, with that build and with Qwen's FP8 (C-Eval / MMLU / GSM8K 82.8 / 84.3 / 91.8 vs 82.8 / 84.5 / 92.2 for that build on the same card; benchmarks). With vLLM 0.31 / 0.32 and --prefix-match-unit 16, TTFT is 25 / 22 ms on an RTX 6000 Ada.

Why another INT4. Most community INT4 builds of this model quantize only the 256 routed experts and keep the GDN, attention and shared-expert projections and lm_head in BF16. On a 3 B-active MoE those BF16 parts are most of the bytes a decode step reads, so those builds land at ~4.6 ms/token on a 5090. Two builds go further in one direction each; this one quantizes all three (checked at tensor level against each repo's main on 2026-10-07). The speed comes from what is quantized, the calibration data decides where the rounding error goes.

Quick start (vLLM ≥ 0.17; measured on 0.24.0, 0.27.1, 0.30.0, 0.31.0, 0.32.0)

vllm serve Twu31/Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency \
  --served-model-name qwen36 --max-model-len 12288 --max-num-seqs 6 \
  --enable-prefix-caching --prefix-match-unit 16 --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

• --mamba-ssm-cache-dtype bfloat16 halves the hybrid model's prefix-cache block (1056 → 544 tokens); outputs were identical with and without it on our harness.
• --prefix-match-unit 16 (measured on vLLM 0.30 to 0.32) also stores the GDN state at the prompt's last 16-token boundary, so a request that shares a fixed system prompt hits within 16 tokens of it instead of at the last 544-token block: TTFT 50 → 25 ms on an RTX 6000 Ada with vLLM 0.31, no prompt padding needed. Drop the flag on vLLM older than 0.26. vLLM 0.32 checks at startup that the unit divides every cache block (544 here, 560 with MTP) and refuses one that does not (vllm#58021); 16 passes.
• MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'): on vLLM 0.24 and 0.27.1 it made the turn slower (below); on 0.30 to 0.32 with --prefix-match-unit 16 it is about even to a small win: for the same reply length, turns 4 to 5 % shorter at 55 to 56 % acceptance on 0.31 (2026-10-07) and on 0.32, and even in a second 0.31 run. Measure on your own replies.
• vLLM 0.32 needs Python 3.11 or newer and pins transformers to 5.16.1–5.19; pip install vllm gets the CUDA 13.0 wheel, and CUDA 12.9 wheels and -cu129 Docker images are on the release page.
• Hopper (H100 / H200 / H20 / H800, SM 9.0): vLLM 0.32 prefers its Humming MoE kernel over Marlin for the routed experts there (vllm#56997); the dense INT4 layers and lm_head use Machete on Hopper, as in earlier versions. Every number on this page is from Ada or Blackwell cards, where 0.32 still picks Marlin; we have not measured Hopper. --moe-backend marlin restores the 0.31 choice for the experts.
• Vision tower is BF16, images work out of the box (--mm-processor-kwargs '{"min_pixels":50176,"max_pixels":200704}' to bound image tokens). Fits a 24 GB card for text; on 32 GB at --gpu-memory-utilization 0.85 vLLM reports 298 k KV tokens. SGLang: untested.
• LoRA (vLLM 0.31 and 0.32): the default --enable-lora also wraps lm_head and fails at start-up on this build's INT8 head (AttributeError reading the head's weight dtype). Pass --lora-target-modules with only the modules your adapter trained, e.g. qkv_proj o_proj in_proj_qkvz out_proj gate_up_proj down_proj. Unmerged outputs matched the same adapter merged into BF16 weights within run-to-run noise (rechecked on 0.32: 96.8 % top-1 agreement, at this model's own run-to-run floor of about 97 %). Decode cost at rank 16, batch 1, RTX 6000 Ada: attention only +5 %, shared expert +9 %, GDN +13 %, all three +26 % (+23 % on 0.32); it depends on which modules are wrapped, not on what the adapter covers.

Latency (measured)

Same harness for every row: a ~1.4 k-token fixed persona system prompt + 2 turns of history + 18 real spoken-Chinese user turns × 3, greedy, streaming, thinking off; medians of 54 requests; replies 20–27 tokens. One RTX 5090 32 GB, vLLM 0.27.1, flags as above without --prefix-match-unit. The anchor is QuantTrio/Qwen3.6-35B-A3B-AWQ (data-free, experts only, AWQ format), a widely used community INT4 build.

vLLM 0.24.0 on the same card: this build 3.2–3.3 ms/token, anchor 4.7. Five concurrent streams (MTP off): this build 329 ms/turn, 9.4 ms/token, 424 tok/s aggregate vs anchor 348 / 12.2 / 321.

MTP did not pay on vLLM 0.27.1: the draft head is a full BF16 MoE decoder layer, the INT4 main body is cheap, replies are 20–30 tokens and acceptance is 53–57 % — k=2 buys ~10 % on ms/token and gives it back in overhead, so the whole turn is slower (159 vs 140 ms). On vLLM 0.30 to 0.32 with --prefix-match-unit 16 it is about even to a small win (next table).

vLLM 0.30 to 0.32 on an RTX 6000 Ada 48 GB (same harness, main; this card has about half the 5090's memory bandwidth, so ms/token is higher than above; KV cache fixed at 10 GiB; 0.30.0 and 0.31.0 run back to back on 2026-10-07, 0.31.0 and 0.32.0 on 2026-10-23; TTFT = median of 54 requests, warm turn = median of the second and third pass, prefix already cached)

Reply lengths differ between rows because greedy replies are not bit-stable across restarts: two restarts of the same vLLM version gave a different reply on 9 to 12 of the 18 prompts on this MoE (kernels are picked by timing at startup), so compare per token: for a 24-token reply the MTP recipe comes to about 114 ms against 120 ms without it on 0.31.0. 0.31.0 is equal to or slightly faster than 0.30.0 in every row. On 0.32.0 the quick start's prefix-cache hit rate on this harness went from 89 % to 97 % against a 0.31.0 run right before it (same requests), which is where its 3 ms lower TTFT comes from; with MTP the hit rate is 77 % on both, and decode speed is unchanged. For a 24-token reply on 0.32.0 the MTP recipe comes to about 114 ms against 119 ms without it; the 0.31.0 run right before it came out even (about 120 against 119 ms). MoE weights run on vLLM's Marlin WNA16 MoE backend on all three versions (on this Ada card; for Hopper see the quick-start notes).

Prefix caching on a hybrid GDN MoE reuses whole blocks only — 1056 tokens with the fp32 recurrent state, 544 with --mamba-ssm-cache-dtype bfloat16 — so pad a fixed system prompt to a block multiple if TTFT matters (on a 5090 the difference is small, 53–63 ms either way; on slower cards it is not). On vLLM 0.26 and newer --prefix-match-unit 16 replaces the padding.

Fidelity vs the BF16 base

Every build is scored token by token against the BF16 base with vLLM 0.31 prompt_logprobs (top 20) on 214,135 positions: 100 zh-Wikipedia and 100 en-WikiText-2 chunks of 1,024 tokens, plus chat — a fixed ~1.5 k-token persona system prompt, 4 turns of history and 450 synthetic spoken-Chinese user turns (written for this test, not in our calibration data), answered greedily by the BF16 base (15,735 answer tokens scored). top-1 = the build's most likely next token is BF16's; KL = KL(BF16 ‖ build) over BF16's top 20 (a lower bound); dNLL = extra nats per token on BF16's own answers. The same model run twice on vLLM agrees on only ~97 % of tokens (KL ≈ 0.003), because batch composition changes the MoE kernels' numerics; differences below that are noise. RTX 6000 Ada, 2026-10-07/08.

The speed has a price here: quantizing the GDN, attention and shared-expert projections as well puts this build further from BF16 than the experts-only builds, at about 3× QuantTrio's KL on wiki text and 2× on chat. Chat, which it was calibrated on, is where it is closest (level with the Intel build, with a lower dNLL). The next table shows which modules the distance comes from and what each would cost to restore; the benchmarks after it show whether it costs task accuracy.

Where the error comes from

Each row puts one class of modules back to BF16 inside this build (spliced from the base, not re-quantized) and repeats the scoring above. Decode = batch-1 ms/token on the same RTX 6000 Ada, vLLM 0.31.0, CUDA graphs, 1,486-token persona prompt, MTP off.

• The INT8 lm_head costs nothing measurable; a BF16 head makes decode 13 % slower.
• The GDN projections are the largest single source of error and also where most of the speed comes from: restoring them costs 40 %.
• The shared expert is the cheapest to restore: 10 % less KL for 3 % more decode time.
• If the parts add up, about half of the KL is the routed experts' GPTQ g128 itself, the same level as the palmfuture GPTQ build and behind the AWQ builds. That share is inferred, not measured on its own, and splicing is approximate: GPTQ fits each layer to the already-quantized layers before it.

Benchmarks

Public test sets, no system prompt, thinking off, greedy. C-Eval validation (all 52 subjects, 1,346 questions); MMLU test, 20 random questions per subject (1,140); GSM8K test, first 500 problems, answered with step-by-step reasoning. Multiple choice is scored by the most likely of the letters A–D at the first answer token. BF16 (72 GB) does not fit the 48 GB card, so its row is one RTX PRO 6000 (96 GB) run on the same items; this build was run on both cards to show the cross-card spread.

On these benchmarks this build is level with QuantTrio and FP8. Every build, FP8 included, lands within about 2 points of the others on each benchmark, about two standard errors at these sample sizes (one standard error is about 1 point on C-Eval and MMLU and 1.2 on GSM8K), so the benchmarks do not separate the builds and the distribution gap in the first table does not show up as a measurable accuracy gap. cyankiwi's g32 build is at or near the top on all three; this build is at the low end on MMLU and GSM8K, within that noise. The same build run on the two cards differs by up to 0.3 points on multiple choice and 0.2 on GSM8K.

Earlier versions of this card used an 18-prompt check in this section (NLL on the BF16 base's answers, exact matches, format probes). Every build passed it, but it could not tell the builds apart, so it has been replaced by the measurements above.

What is quantized, what is not

Format: compressed-tensors pack-quantized, group_0 = INT4 g128 sym actorder=weight on Linear, group_1 = INT8 g128 on re:.*lm_head$; per-expert tensors (experts.N.gate_proj.weight_packed, vLLM CompressedTensorsWNA16MarlinMoEMethod). 21.5 GB on disk (BF16 base 71.9 GB). Until 2026-10-07 the ignore list in config.json also carried 60 entries that matched no tensor, layers.N.linear_attn and layers.N.linear_attn.norm: the compressed-tensors 0.18 exporter treated any class whose name contains Gate as a Linear layer, which caught GatedDeltaNet and RMSNormGated (fixed upstream in compressed-tensors#871). They were no-ops, since the GDN projections listed above are INT4 and vLLM never looks those names up, and they have been removed from config.json; weights are unchanged.

Calibration & reproduce

384 sequences ≥ 2048 tokens, rendered through the model's own chat template with enable_thinking=False: ~60 % multi-turn spoken-dialogue transcripts (a ≈1.4 k-token assistant-persona system prompt + 10–20 short colloquial turns, predominantly Chinese, real speech incl. disfluencies), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. moe_calibrate_all_experts=True so rarely-routed experts still get a usable Hessian.

llm-compressor 0.13.0 · transformers 5.14.1 · compressed-tensors 0.18.0 · torch 2.13; one RTX 5090 + 120 GB RAM, 3 h wall-clock (GPTQ's per-module loop over 256 experts × 3 × 40 layers; independent of sample count).

import torch
from datasets import Dataset
from llmcompressor import oneshot
from llmcompressor.modeling.moe.linearize import load_quantizable_moe
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration

MODEL = "Qwen/Qwen3.6-35B-A3B"
with load_quantizable_moe(Qwen3_5MoeForConditionalGeneration):          # fused 3-D experts -> per-expert nn.Linear
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(MODEL)
ds = Dataset.from_list([{"text": t} for t in pre_rendered_texts])       # 384 x >=2048 tokens, chat template applied, thinking off
ignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$",
          r"re:.*mlp\.gate$", r"re:.*shared_expert_gate$"]
recipe = GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")
oneshot(model=model, processor=tok, dataset=ds, recipe=recipe, max_seq_length=2048,
        num_calibration_samples=384, moe_calibrate_all_experts=True)
model.save_pretrained("out", save_compressed=True, max_shard_size="4GB"); tok.save_pretrained("out")

Three things the exporter does not do for you: (1) copy the BF16 mtp.* tensors from the base checkpoint into the export and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads packed weights it cannot use and MTP runs at 0 %; (2) apply the INT8 lm_head post-hoc (compressed-tensors packer, group_1, target re:.*lm_head$); (3) copy the upstream tokenizer / config files back verbatim (transformers 5.14 re-serialises tokenizer.json). Full scripts (quantization, export fix-up, INT8 head, evaluation harness) are in our repo, llm_quant/qwen36_35b_zhchat/.

Limitations

• Built and measured for short-reply, thinking-off, single/few-stream spoken chat. Long chain-of-thought (the benchmarks above run with thinking off), tool use and > 12 k context were not evaluated. Fidelity is measured on wiki text and synthetic chat, not on your traffic.
• Measured on one RTX 5090 (vLLM 0.24.0 / 0.27.1), one RTX 6000 Ada (vLLM 0.30.0 / 0.31.0 / 0.32.0) and, for the BF16 benchmark row, one RTX PRO 6000 (vLLM 0.31.0); the Marlin path is the same on Ampere and Ada, absolute numbers follow memory bandwidth.
• vLLM 0.32 dropped Python 3.10. vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.6; the vLLM / llm-compressor / compressed-tensors maintainers.