Model Overview

  • Model Architecture: Qwen3_5MoeForConditionalGeneration
    • Input: Text, Image, Video
    • Output: Text
  • Supported Hardware Microarchitecture: AMD MI350 / MI355
  • ROCm: 7.2.4
  • PyTorch: 2.10.0+rocm7.2.4
  • Transformers: 5.15.1
  • Operating System(s): Linux
  • Inference Engine: SGLang
  • Model Optimizer: AMD-Quark (v0.13)
    • Quantized layers: All MoE experts in the language model, including the shared expert (the shared expert is also fused into the MoE kernel for faster decode). MTP routed experts and the MTP shared expert are quantized the same way. MTP self-attention is quantized to FP8.
    • Weight quantization:
      • self_attn and linear_attn Perchannel, FP8E4M3, Static
      • MOE OCP MXFP4, Static
    • Activation quantization:
      • self_attn and linear_attn Pertoken, FP8E4M3, Dynamic
      • MOE OCP MXFP4, Dynamic

This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert), and by quantizing the MTP experts to MXFP4 and the MTP self-attention to FP8.

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

MODEL_PATH="Qwen/Qwen3.5-397B-A17B"
OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3"

cd /path/to/Quark

python3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir "${MODEL_PATH}" \
  --quant_scheme mxfp4 \
  --layer_quant_scheme '*linear_attn*' ptpc_fp8 \
  --layer_quant_scheme '*self_attn*' ptpc_fp8 \
  --quant_flow file2file \
  --no_trust_remote_code \
  --model_export hf_format \
  --skip_evaluation \
  --multi_gpu balanced \
  --exclude_layers "lm_head" "*mlp.gate" "mtp.fc" "model.visual.*" "*shared_expert_gate*" "*linear_attn.conv1d*" \
  --output_dir "${OUTPUT_DIR}"

For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.

Evaluation

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B checkpoint, evaluated with the identical recipe on SGLang.

This model can be deployed efficiently using the SGLang backend based on the Docker image lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260922.

Accuracy

Benchmark Qwen/Qwen3.5-397B-A17B amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 (this model) Recovery
gsm8k (flexible-extract, 5-shot) 97.71 97.57 99.86%

Reproduction

The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe. Both sides were served with EAGLE MTP.

  1. Serve the model with SGLang:
export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export SGLANG_USE_AITER_FP8_PER_TOKEN=1

python3 -m sglang.launch_server \
    --model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 4 \
    --attention-backend aiter \
    --mem-fraction-static 0.90 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --watchdog-timeout 1200 \
    --disable-radix-cache \
    --reasoning-parser qwen3 \
    --page-size 16 \
    --enable-aiter-allreduce-fusion \
    --enable-metrics \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4
  1. Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy). This uses InferenceX's gsm8k.yaml, whose only change vs. the stock lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:
lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k.yaml \
  --num_fewshot 5 \
  --model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=16,tokenized_requests=False,max_length=16384" \
  --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.