Qwen3.6-35B-A3B-INT4-RTN

INT4 (weight-only) quantization of Qwen/Qwen3.6-35B-A3B, produced with llm-compressor using RTN (round-to-nearest, data-free) quantization.

The quantization scheme is a faithful reproduction of the native INT4 scheme used by Kimi-K2.6 / Kimi-K2-Thinking: INT4 weight-only, group size 32, symmetric, applied only to the routed MoE experts and stored in the compressed-tensors pack-quantized format.

Note: Moonshot produced Kimi's weights with Quantization-Aware Training (QAT). This checkpoint reproduces the same scheme and on-disk format via post-training RTN — not QAT — so it is directly loadable by the same inference engines, but is not QAT-trained.

Quantization details

This mirrors Kimi's config: num_bits=4, type=int, strategy=group, group_size=32, symmetric=true, format=pack-quantized, quantizing the MoE components while keeping attention and the shared expert at full precision.

Result: ~72 GB (BF16) → ~23 GB on disk.

Usage (vLLM)

vllm serve casperhansen/Qwen3.6-35B-A3B-INT4-RTN \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --reasoning-parser qwen3
# add --language-model-only for text-only serving

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="casperhansen/Qwen3.6-35B-A3B-INT4-RTN",
    messages=[{"role": "user", "content": "Give me a fun fact about the ocean."}],
    max_tokens=256, temperature=0.7, top_p=0.8,
)
print(resp.choices[0].message.content)

Verified to load and generate on a single NVIDIA H200 with vLLM (INT4 WNA16 MoE Marlin path).

Reproduction

from compressed_tensors.config import CompressionFormat
from compressed_tensors.quantization import (
    QuantizationArgs, QuantizationScheme, QuantizationStrategy, QuantizationType,
)
from llmcompressor import model_free_ptq

scheme = QuantizationScheme(
    targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
    weights=QuantizationArgs(
        num_bits=4, type=QuantizationType.INT, strategy=QuantizationStrategy.GROUP,
        group_size=32, symmetric=True, observer="minmax", dynamic=False,
    ),
    format=CompressionFormat.pack_quantized.value,
)

model_free_ptq(
    model_stub="Qwen/Qwen3.6-35B-A3B",
    save_directory="Qwen3.6-35B-A3B-INT4-RTN",
    scheme=scheme,
    max_workers=8,
    device="cuda:0",
)

License

Apache-2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B.