Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0

Model Overview

• Model Architecture: Qwen3_5MoeForConditionalGeneration
  • Input: Text
  • Output: Text
• Source Model: Qwen3.6-35B-A3B
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.28.0
• Quantization Framework: LLM Compressor v0.12.0
• Quantization Method: 4-bit Weight-Only Quantization (W4A16)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.13.0.0
  • PyTorch v2.13.0.0
  • LLM Compressor v0.12.0
  • vLLM v0.28.0

This is a quantized version of Qwen3.6-35B-A3B created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Qwen3.6-35B-A3B using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 67.0 GiB to 18.1 GiB on disk (~73% reduction).

• Method: 4-bit Weight-Only Quantization (W4A16)
• Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128
• Weights: INT4, symmetric, group-wise (group_size=128, actorder=static), stored as pack-quantized
• Activations: BF16 (unquantized)
• Group Size: 128, valid here because every quantized weight's column count (2048 hidden, 512 MoE intermediate) divides by 128
• Calibration: 128 examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048
• Kept in BF16: the vision tower (model.visual), the MoE router (mlp.gate), shared_expert_gate, lm_head, embed_tokens, and the layer norms. The 256 routed experts, the shared expert, and the linear-attention projections are all quantized, which is why the reduction is close to the 75% ideal for INT4.

Three details in the recipe are specific to this model. The model is loaded with AutoModelForImageTextToText rather than AutoModelForCausalLM, so vision_config and the Qwen3_5MoeForConditionalGeneration metadata survive into the saved config for vLLM 0.26+; without it the config is demoted to the text-only variant and vLLM rejects the checkpoint. shared_expert_gate is skipped because it has out_features=1 and vLLM expects a dense .weight there rather than a packed one. And the router is skipped because it is a tiny Linear whose logits decide expert assignment, where a mis-routed token costs far more accuracy than 4-bit expert weights do.

import json
import os

import torch
from datasets import load_dataset
from transformers import AutoModelForImageTextToText, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization.gptq import GPTQModifier

MODEL_ID = "Qwen/Qwen3.6-35B-A3B"
OUTPUT_DIR = "./Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0"
CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

IGNORE = [
    "lm_head",
    r"re:.*\.router$",
    r"re:.*\.router\..*",
    r"re:.*\.gate$",
    r"re:.*\.mlp\.gate$",
    "shared_expert_gate",
    r"re:.*\.shared_expert_gate(\.|$)",
    r"re:.*\.visual(\.|$)",
    r"re:.*\.visual\..*",
]

# Step 1: Load the BF16 model with the multimodal wrapper so vision_config and
# the ForConditionalGeneration architecture are preserved on save.
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
reference_config = model.config
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations
# to build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))},
    remove_columns=ds.column_names,
)
if not getattr(tokenizer, "pad_token", None):
    tokenizer.pad_token = tokenizer.eos_token
calib_ds = ds.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False
    ),
    remove_columns=["text"],
)

# Step 3: Define the W4A16 GPTQ recipe and one-shot quantize. Passing
# processor=MODEL_ID saves the processor configs alongside the model.
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=IGNORE)
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    processor=MODEL_ID,
    output_dir=OUTPUT_DIR,
    save_compressed=True,
    trust_remote_code_model=True,
)

# Step 4: Patch config.json so architectures/vision_config stay intact for vLLM.
config_path = os.path.join(OUTPUT_DIR, "config.json")
with open(config_path, encoding="utf-8") as f:
    saved = json.load(f)
full = reference_config.to_dict()
for key in (
    "architectures",
    "model_type",
    "text_config",
    "vision_config",
    "vision_start_token_id",
    "vision_end_token_id",
    "image_token_id",
    "video_token_id",
):
    if full.get(key) is not None:
        saved[key] = full[key]
with open(config_path, "w", encoding="utf-8") as f:
    json.dump(saved, f, indent=2)
    f.write("\n")

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Run end to end with the standalone script:

numactl --physcpubind=0-95 python quant-qwen3.6-w4a16.py

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0.0
zentorch==2.13.0.0
vllm==0.28.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0,tokenizer=Qwen/Qwen3.6-35B-A3B,dtype=bfloat16,max_model_len=4096,language_model_only=True,enable_thinking=False \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0.0 / PyTorch v2.13.0.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Vision Path Unquantized: The vision tower stays in BF16, so image preprocessing cost is unchanged and the memory saving comes entirely from the language model. Evaluation was run with language_model_only=True.
• Reasoning Model: Qwen3.6 emits thinking traces by default. Evaluate with enable_thinking=False (and a generous max_gen_toks) so answer extraction stays comparable to the BF16 baseline.
• Accuracy Trade-off: 4-bit weight-only quantization is more aggressive than INT8 and trades some accuracy for a ~73% smaller memory footprint.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.