Model Overview

Model Architecture: MuseSparkForConditionalGeneration
- Input: Text, Image, Video, Audio
- Output: Text
Supported Hardware Microarchitecture: AMD MI350/MI355
Operating System(s): Linux
Inference Engine: SGLang / vLLM
Model Optimizer: AMD-Quark(v0.12)
- Quantized layers: MoE experts, except layer 0 and layer 61.
- Weight quantization:
  - MOE OCP MXFP4, Static
- Activation quantization:
  - MOE OCP MXFP4, Dynamic

This checkpoint quantizes the routed MoE experts to OCP MXFP4 (except those in layers 0 and 61) and keeps everything else in bf16.

Model Quantization

The model was quantized from meta-models/Muse-Spark-Open-816B-A42B using AMD-Quark. The weights and activations are quantized to MXFP4.

Quantization scripts:

from quark.torch import ModelQuantizer
from quark.torch.quantization.config.template import LLMTemplate
from quark.torch.quantization.weight_convert import SplitFusedExperts, WeightConverter

LLMTemplate.register_template(
    LLMTemplate(
        model_type="muse_spark",
        kv_layers_name=["*k_proj", "*v_proj"],
        q_layer_name="*q_proj",
        exclude_layers_name=[
            "lm_head",
            "*self_attn*",
            "*mlp.gate",
            "*pre_expert_proj",
            "*post_expert_proj",
            "*vision*",
            "*audio*",
            "*language_model.layers.0.*",
            "*language_model.layers.61.*",
        ],
        f2f_weight_converters=[
            WeightConverter(
                "gate_up_proj",
                ["gate_proj.weight", "up_proj.weight"],
                operations=[SplitFusedExperts(split_axis=0)],
            ),
            WeightConverter(
                "down_proj",
                ["down_proj.weight"],
                operations=[SplitFusedExperts(split_axis=0)],
            ),
        ],
    )
)

template = LLMTemplate.get("muse_spark")
quant_config = template.get_config(scheme="mxfp4")

quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path="meta-models/Muse-Spark-Open-816B-A42B",
    save_path="amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4",
    weight_converters=template.f2f_weight_converters,
)

Deployment

The following ROCm recipes serve this checkpoint on AMD Instinct MI355X GPUs. The SGLang recipe is tuned for low latency with DFlash V2 speculative decoding; the vLLM recipe uses tensor parallelism and serves the language model path. For the latest flags and alternative strategies, see the SGLang cookbook and vLLM recipe.

SGLang: MI355X low-latency recipe

The selected recipe uses TP4 + EP4, AITER attention, FlyDSL MoE, and the RadixArk/Muse-Spark-Open-DFlash2 draft model.

docker pull lmsysorg/sglang-rocm:muse-spark-open

docker run \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env SGLANG_USE_AITER=1 \
  --env SGLANG_USE_AITER_UNIFIED_ATTN=1 \
  --env MUSE_MOE_BACKEND=flydsl \
  --ipc=host \
  lmsysorg/sglang-rocm:muse-spark-open \
  sglang serve \
  --model-path amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 \
  --tp-size 4 \
  --ep-size 4 \
  --attention-backend aiter \
  --page-size 16 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
  --speculative-draft-attention-backend triton \
  --speculative-num-draft-tokens 8 \
  --reasoning-parser muse \
  --tool-call-parser muse

vLLM: MI355X MXFP4 recipe

The selected recipe uses TP8, AITER kernels, chunked prefill, and a 32,768-token serving limit. --language-model-only configures the ROCm path for text serving.

docker pull vllm/vllm-openai-rocm:muse-spark-open

docker run --device=/dev/kfd --device=/dev/dri \
  --security-opt seccomp=unconfined --group-add video \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai-rocm:muse-spark-open \
  amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --distributed-executor-backend mp \
  --language-model-only \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --attention-backend ROCM_AITER_FA \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser muse \
  --enable-auto-tool-choice \
  --tool-call-parser muse

Send a request

Use port 30000 for the SGLang command or port 8000 for the vLLM command.

curl http://localhost:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4",
    "messages": [{"role": "user", "content": "Write a short poem about AMD Instinct GPUs."}],
    "reasoning_effort": "high"
  }'

If authentication is required in your environment, pass HF_TOKEN to the container through an environment variable rather than embedding the token in the command.

For additional background, see the SGLang AMD support announcement.

amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4