Qwen3.6-35B-A3B-MXFP8-FP8KV-CT-AutoRound

An MXFP8 checkpoint of Qwen3.6-35B-A3B, with FP8 KV-cache support.

Model Details

The routed-expert weights use E4M3 with a group size of 32. The export stores their group scales in U8 tensors. Attention, linear-attention, shared-expert, router, vision, embedding, language-model-head, and MTP weights are retained at BF16 where applicable; FP32 scale and auxiliary tensors retain their checkpoint dtypes.

Quantization Recipe

Scope and settings

This recipe converts the fused routed-expert weights to MXFP8 using RTN with zero optimization iterations, and writes static FP8 KV-cache metadata. It leaves the following module families out of weight quantization: lm_head, embed_tokens, visual, linear_attn, shared_expert, self_attn, mtp, and mlp.gate. Fused routed experts are handled by the model's MoE export path.

The command below reproduces the settings used for this checkpoint. Replace the input and output paths with paths available on your host. The example assigns three visible GPUs; change CUDA_VISIBLE_DEVICES and --device_map together to match the devices allocated to the process.

export CUDA_VISIBLE_DEVICES=0,1,2

auto-round \
  --model_name /path/to/Qwen3.6-35B-A3B \
  --scheme MXFP8 \
  --algorithm rtn \
  --iters 0 \
  --static_kv_dtype fp8 \
  --ignore_layers lm_head,embed_tokens,visual,linear_attn,shared_expert,self_attn,mtp,mlp.gate \
  --device_map 0,1,2 \
  --format llm_compressor \
  --output_dir /path/to/output \
  --disable_low_cpu_mem_usage

Export checks

Before serving a new export, verify the safetensors index against the shard headers and inspect the actual tensor dtypes. This checkpoint contains 63,191 indexed tensor keys with no index/header mismatch. Its routed experts comprise 30,720 F8_E4M3 weight tensors and 30,720 U8 scale tensors; the remaining indexed tensors are BF16 or FP32. The published snapshot is approximately 40.73 GB across 20 files.

For this run, the tested toolchain was AutoRound 0.16.0.dev193+g22d6e28c, PyTorch 2.13.0+cu130, Transformers 5.17.0, vLLM 0.29.0, and lm-evaluation-harness 0.4.13.

Inference Recipe

The command below was validated on one host with four GPUs, using tensor parallelism, data parallelism, expert parallelism, and FP8 KV cache:

export CUDA_VISIBLE_DEVICES=0,1,2,3

vllm serve intel-ai/Qwen3.6-35B-A3B-MXFP8-FP8KV-CT-AutoRound \
  --tokenizer Qwen/Qwen3.6-35B-A3B \
  --host 127.0.0.1 \
  --port 8000 \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --tensor-parallel-size 2 \
  --data-parallel-size 2 \
  --data-parallel-size-local 2 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90

Adjust the parallel topology, visible devices, memory utilization, and context limit to fit the available hardware. The example reflects the tested serving configuration; it is not a requirement to use four GPUs for every deployment.

Evaluation Recipe

The benchmark results below were measured with the repository-defined GSM8K, PIQA, MMLU, and HellaSwag tasks, batch size 32, seed 42, and a 131,072-token client-side length limit. Start the inference server above before running the commands.

For GSM8K, use the following generation settings. The end-of-turn token is the stop condition so that a Question: string inside a thinking trace does not prematurely stop generation.

lm_eval \
  --model local-completions \
  --model_args "model=intel-ai/Qwen3.6-35B-A3B-MXFP8-FP8KV-CT-AutoRound,tokenizer=Qwen/Qwen3.6-35B-A3B,tokenizer_backend=huggingface,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=True,num_concurrent=1,max_retries=50,timeout=500,max_length=131072,max_gen_toks=2048" \
  --tasks gsm8k \
  --batch_size 32 \
  --output_path ./results/gsm8k \
  --log_samples \
  --seed 42 \
  --num_fewshot 5 \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --gen_kwargs 'max_gen_toks=2048,until=<|im_end|>'

Run the three log-likelihood tasks with:

lm_eval \
  --model local-completions \
  --model_args "model=intel-ai/Qwen3.6-35B-A3B-MXFP8-FP8KV-CT-AutoRound,tokenizer=Qwen/Qwen3.6-35B-A3B,tokenizer_backend=huggingface,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=True,num_concurrent=1,max_retries=50,timeout=500,max_length=131072,max_gen_toks=2048" \
  --tasks piqa,mmlu,hellaswag \
  --batch_size 32 \
  --output_path ./results/piqa_mmlu_hellaswag \
  --log_samples \
  --seed 42

Benchmark Results

All scores are shown as percentages to two decimal places. Accuracy retained is calculated from the unrounded scores as quantized_model_score / original_model_score * 100%. Relative change is calculated as (quantized_model_score / original_model_score - 1) * 100%; it is a ratio-based relative change, not a percentage-point difference.

The supplementary GSM8K flexible-extract score was 79.98% for the original model and 79.30% for this checkpoint, with 99.15% accuracy retained and a -0.85% relative change. The mean row uses the four primary metrics above, including GSM8K strict-match.

The original BF16 model used its default KV-cache dtype, while this checkpoint was evaluated with FP8 KV cache. The comparison therefore reflects the combined weight-format and KV-cache configuration; it does not isolate weight quantization from KV-cache effects. These four benchmarks are not a 128K long-context stress test.