Qwopus3.6-35B-A3B-v1-int4-mixed

W4A16 mixed-precision quantization of Jackrong/Qwopus3.6-35B-A3B-v1, created with llm-compressor. Native vLLM support via compressed-tensors format.

Only MoE expert layers are quantized to INT4. All attention layers, shared experts, vision encoder, and MTP layers are preserved in BF16 to maintain quality and prevent infinite thinking loops.

Quantization Details

Layers preserved in BF16

To maintain quality and prevent infinite thinking loops, the following layers are kept in full precision:

• self_attn — full attention q/k/v/o projections (layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39)
• linear_attn — all linear attention projections (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj)
• shared_expert — shared expert gate/up/down projections
• shared_expert_gate — shared expert routing
• mlp.gate — MoE router
• embed_tokens — input embeddings
• visual — entire vision encoder
• mtp — multi-token prediction layers
• lm_head — output head

What is quantized

Only the MoE expert layers (256 experts × 3 Linear layers × 40 blocks) are quantized to INT4. These account for ~90% of total parameters but are the least sensitive to quantization.

Benchmark Results

Best scores from 3 evaluation runs using EvalScope with thinking mode enabled.

Evaluation parameters Evaluation framework: EvalScope (evalscope)

Generation parameters (both models):

{
  "max_tokens": 20000,
  "temperature": 1.0,
  "top_p": 0.8,
  "top_k": 20,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0
}

EvalScope command:

OPENAI_TIMEOUT=600 evalscope eval \
  --model <model_name> \
  --api-url http://localhost:8000/v1 \
  --api-key none \
  --datasets gpqa_diamond humaneval gsm8k ifeval \
  --eval-batch-size 8 \
  --generation-config '{"max_tokens": 20000, "temperature": 1.0, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5, "repetition_penalty": 1.0}'

Note: presence_penalty=1.5 is Qwen's official recommendation for thinking mode to prevent infinite thinking loops. Scores vary significantly with different generation parameters — results without presence_penalty are substantially lower for both bf16 and INT4.

Usage

vLLM (recommended)

vllm serve Avesed/Qwopus3.6-35B-A3B-v1-int4-mixed \
  --tokenizer Qwen/Qwen3.6-35B-A3B \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --trust-remote-code \
  --generation-config vllm \
  --override-generation-config '{"temperature": 1.0, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5, "repetition_penalty": 1.0, "max_tokens": 65536}'

Sampling parameters

Qwen3.6 in thinking mode benefits from presence_penalty=1.5 to prevent infinite thinking loops. The generation config above reflects Qwen's official recommendations.

Acknowledgements

• Original model by Jackrong
• Original base model by Qwen