Qwen3.6-27B — W8A16 (INT8 weights, FP16 activations)

Overview

Qwen3.6-27B-W8A16-AutoRound is an 8-bit checkpoint designed to reduce inference memory requirements, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.safetensors (11 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• processor_config.json
• chat_template.jinja
• quantization_config.json
• Additional configuration, tokenizer, processor, or shard files (20 visible artifacts total)

Quick start

Runtime selection

Load the checkpoint with a runtime that supports its architecture and 8-bit weight format. Keep the repository's configuration and tokenizer/processor files together with the weight files.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

INT8 weight-only quantization of Qwen/Qwen3.6-27B, produced with Intel AutoRound and exported in the compressed-tensors (pack-quantized) format for direct loading in vLLM. The MTP (multi-token-prediction) head is shipped and works for speculative decoding — see below.

Quantization details

The language-model Linear layers are quantized to INT8 with group size 128. The visual encoder blocks and the MTP head are kept at full precision, so the checkpoint stays compatible with both multimodal and speculative-decoding paths in vLLM.

Usage (vLLM)

vllm serve groxaxo/Qwen3.6-27B-W8A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --dtype half \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

MTP speculative decoding

vllm serve groxaxo/Qwen3.6-27B-W8A16-AutoRound \
  --tensor-parallel-size 2 --dtype half \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Observed on 2× RTX 3090 with this checkpoint, draft acceptance is workload-dependent: ~80–90 % on factual / repetitive text, dropping to ~60–70 % on diverse generation, with mean acceptance length ≈ 2.2–2.8 (i.e. ~2–3 tokens per decode step). Actual decode speedup depends on batch size and prompt; benchmark on your own hardware.

Important: MTP requires the mtp.* layers in the quant ignore list

AutoRound leaves the mtp.* layers unquantized but does not add them to the quantization_config.ignore list. Without that, vLLM builds the MTP drafter as a W8A16 (Marlin) module, fails to match the plain-FP16 checkpoint weights (Parameter ... not found in params_dict, skip loading), and the drafter runs with garbage weights → 0 % acceptance.

This repo already includes the fix: the 8 unfused MTP Linear names are present in the ignore list of both config.json and quantization_config.json:

mtp.fc
mtp.layers.0.self_attn.{q,k,v,o}_proj
mtp.layers.0.mlp.{gate,up,down}_proj

(compressed-tensors un-fuses qkv_proj/gate_up_proj, so the individual shard names are listed.) If you re-quantize this model yourself, apply the same edit. Note that vLLM's torch.compile cache is not invalidated by per-layer quant-scheme changes — if you change the ignore list on an already-booted model, rm -rf ~/.cache/vllm/torch_compile_cache before restarting.

Notes

W8A16 roughly halves the weight footprint versus BF16 (~30 GB vs ~54 GB) while keeping activations in FP16. Ampere GPUs (e.g. RTX 3090) support INT8 weight-only kernels natively. Calibration-free RTN was used; for the smallest possible accuracy gap you can re-quantize with the iterative AutoRound algorithm and a calibration dataset.