Muse-Spark-Open-816B-A42B · MXFP4 (AutoRound)

License: Muse Model License

This is an MXFP4 (4-bit OCP Micro-scaling FP4, weight + dynamic activation) quantized build of Muse-Spark-open-816B-A42B (a MuseSparkForConditionalGeneration MoE model: 62 decoder layers, 256 experts, 8-of-256 per token, hidden_size=8192, moe_intermediate_size=4096, 0 shared experts), exported by AutoRound with --format auto_round (auto_round:llm_compressor packing).

Only the MoE expert projections of the middle layers (L1–L60) are quantized to MXFP4; the attention block, the MoE router mlp.gate, the per-layer pre_expert_proj / post_expert_proj, the first (L0) and last (L61) layers' experts, and all multimodal towers are kept in full precision (BF16; router in FP32).

Overview

Key fields of quantization_config (see config.json):

{
  "quant_method": "auto-round",
  "packing_format": "auto_round:llm_compressor",
  "data_type": "mx_fp",
  "bits": 4,
  "group_size": 32,
  "sym": true,
  "act_bits": 4,
  "act_data_type": "mx_fp",
  "act_dynamic": true,
  "act_group_size": 32,
  "act_sym": true,
  "iters": 0,
  "model_free": true,
  "autoround_version": "0.16.0",
  "block_name_to_quantize": "model.language_model.layers",
  "extra_config": { /* 2337 entries kept at bits=16 — see "Kept full-precision" above */ }
}

Every module that must stay full-precision is recorded in extra_config as {"bits": 16, "data_type": "float", "act_bits": 16, "act_data_type": "float"}; its weights therefore ship as plain .weight. Only the quantized MoE expert projections carry weight_packed / weight_scale. The 2337 entries expand to: attention (5 × 62) + router mlp.gate (62) + pre/post_expert_proj (2 × 62) + vision Linear (6 × 50) + L0 & L61 experts (3 × 256 × 2 = 1536) + singles (lm_head, vision_adapter.fc1/fc2, vision_projection, audio_tower.proj).

Quantization command

export CUDA_VISIBLE_DEVICES=2

AR_MODEL_FREE_SHARD_PARALLELISM=1 auto-round \
  --model_name meta-models/Muse-Spark-Open-816B-A42B \
  --scheme MXFP4 \
  --model_free \
  --disable_torch_compile \
  --ignore_layers "self_attn,pre_expert_proj,post_expert_proj,mlp.gate,vision_tower,vision_adapter,vision_projection,audio_tower,embed_tokens,lm_head,model.language_model.layers.0.mlp.experts,model.language_model.layers.61.mlp.experts" \
  --device_map auto \
  --format auto_round \
  --output_dir ./Muse-Spark-Open-816B-A42B-MXFP4

Notes:

• --ignore_layers uses substring matching against the full module name (if fp_layer in name). The two trailing entries model.language_model.layers.0.mlp.experts / ...layers.61.mlp.experts keep only the first and last layers' experts in BF16 (verified: they match exactly L0 & L61, 768 modules each, with no collateral on layers.10/20/... or the towers).
• Patterns that end in a digit get a . auto-appended by AutoRound (layers.6 → layers.6.), so prefer the explicit .mlp.experts suffix used above to avoid layers.6 matching layers.60.
• Keep the comma list clean — an empty item (e.g. an accidental double comma) makes AutoRound run name[-1] on an empty string and crash with IndexError during layer-config resolution.
• Unlike the weight-only W4A16 build, MXFP4 additionally applies dynamic MXFP4 activation quantization (act_bits=4, 32-element blocks) — i.e. W4A4.

What is quantized (verified tensor-level)

Tensor census (65 shards, 95,572 tensors): weight_packed = 46,080 and weight_scale = 46,080 (= 60 layers × 256 experts × 3 projections), i.e. exactly the L1–L60 experts; 0 weight_packed tensors live outside *.experts.*, and there are 0 global_scale / zero_point tensors (pure MX: only the E8M0 block scale, no NVFP4-style per-tensor global scale). Everything else — attention (self_attn.*, 310 BF16), router (mlp.gate, 124 FP32 tensors), pre/post_expert_proj (62 + 62 BF16), all norms & residual gates, the full vision tower (806 BF16) and audio tower (72 BF16), the multimodal projectors, embed_tokens and lm_head — ships in full precision.

Usage (vLLM)

Recommended inference parameters

Inherited from the base model (Muse-Spark-Open-816B-A42B) — quantization does not change them:

• Sampling: temperature = 1.0, top_p = 1.0, top_k disabled.
• Reasoning effort: set reasoning_effort to low / medium / high / xhigh (default xhigh); use high or xhigh for complex reasoning, coding, and agentic tasks, and low / medium for faster, shorter responses. Parsed by reasoning_parser=musespark1x.

Launch

Key launch parameters (grounded in the evaluation harness below):

• reasoning_parser=musespark1x — natively registered in vllm/reasoning/__init__.py.
• tokenizer_mode=auto — the checkpoint ships a standard PreTrainedTokenizerFast; do not pass the old musespark1x tokenizer mode (it is no longer registered and will error).
• dtype=bfloat16, config_format=auto, load_format=auto.
• Env: VLLM_WORKER_MULTIPROC_METHOD=spawn, FLASHINFER_DISABLE_VERSION_CHECK=1, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, TRANSFORMERS_OFFLINE=1.

Equivalent vllm serve (tensor-parallel size is deployment-dependent — ≈ 474 GB of weights needs ≥ 2× H200/B300 class GPUs; MXFP4 W4A4 inference benefits from OCP-MX FP4 Tensor Cores):

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export FLASHINFER_DISABLE_VERSION_CHECK=1
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

vllm serve intel-ai/Muse-Spark-Open-816B-A42B-MXFP4 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --tokenizer-mode auto \
  --reasoning-parser musespark1x \
  --served-model-name Muse-Spark-Open-816B-A42B-MXFP4

Evaluation (lm-eval-harness · vLLM backend)

The following script reproduces the accuracy evaluation on piqa / mmlu / hellaswag and gsm8k:

export CUDA_VISIBLE_DEVICES=1,2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export FLASHINFER_DISABLE_VERSION_CHECK=1
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export TRANSFORMERS_OFFLINE=1

NUM_GPUS=2
GPU_MEMORY_UTILIZATION=${GPU_MEMORY_UTILIZATION:-0.9}

MODEL=${MODEL:-intel-ai/Muse-Spark-Open-816B-A42B-MXFP4}
MODEL_ARGS="pretrained=${MODEL},tensor_parallel_size=${NUM_GPUS},max_model_len=8192,max_num_batched_tokens=4096,max_num_seqs=16,add_bos_token=True,gpu_memory_utilization=0.95,dtype=bfloat16,max_gen_toks=2048,enable_prefix_caching=False,reasoning_parser=musespark1x,tokenizer_mode=auto,config_format=auto,load_format=auto,enable_thinking=False"

# Multiple-choice / likelihood tasks
lm_eval \
  --model vllm \
  --model_args $MODEL_ARGS \
  --tasks piqa,mmlu,hellaswag \
  --batch_size 128 \
  --seed 42 \
  --output_path lm_eval_results

# Generative task (chat template + multi-turn few-shot)
lm_eval \
  --model vllm \
  --model_args $MODEL_ARGS \
  --tasks gsm8k \
  --batch_size 128 \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --seed 42 \
  --output_path lm_eval_results

Notes:

• tensor_parallel_size / CUDA_VISIBLE_DEVICES are set for a 2-GPU run; scale them to your host.

License & Attribution

Built with Muse Spark Open.

This model is a quantized derivative of Muse-Spark-Open-816B-A42B, released under the Muse Model License (license_name: muse-model-license; full text in LICENSE.md). Per the license, this notice, the license, and its accompanying Usage Policy must be retained in all copies and further derivatives, and products or services built on this model must prominently display "Built with Muse Spark Open".