Muse-Spark-Open-816B-A42B-INT4

License: Muse Model License

This is a W4A16 (weight-only INT4) quantized build of Muse-Spark-Open-816B-A42B (a MuseSparkForConditionalGeneration MoE model: 62 decoder layers, 256 experts, 8-of-256 per token, hidden_size=8192, moe_intermediate_size=4096, 0 shared experts), exported by AutoRound with --format auto_round (auto_round:auto_gptq packing).

Only the MoE expert projections of the middle layers (L1–L60) are quantized to INT4; the attention block, the MoE router mlp.gate, the per-layer pre_expert_proj / post_expert_proj, the first (L0) and last (L61) layers' experts, and all multimodal towers are kept in full precision (BF16; router in FP32). Activations are not quantized (weight-only).

Overview

Key fields of quantization_config (see config.json):

{
  "quant_method": "auto-round",
  "packing_format": "auto_round:auto_gptq",
  "data_type": "int",
  "bits": 4,
  "group_size": 128,
  "sym": true,
  "iters": 0,
  "model_free": true,
  "autoround_version": "0.16.0",
  "block_name_to_quantize": "model.language_model.layers",
  "extra_config": { /* 2337 entries kept at bits=16 — see "Kept full-precision" above */ }
}

Every module that must stay full-precision is recorded in extra_config as {"bits": 16, "data_type": "float"}; its weights therefore ship as plain .weight. Only the quantized MoE expert projections carry qweight / qzeros / scales. The 2337 entries expand to: attention (5 × 62) + router mlp.gate (62) + pre/post_expert_proj (2 × 62) + vision Linear (6 × 50)

• L0 & L61 experts (3 × 256 × 2 = 1536) + singles (lm_head, vision_adapter.fc1/fc2, vision_projection, audio_tower.proj).

Quantization command

export CUDA_VISIBLE_DEVICES=2

AR_MODEL_FREE_SHARD_PARALLELISM=1 auto-round \
  --model_name meta-models/Muse-Spark-Open-816B-A42B \
  --scheme W4A16 \
  --model_free \
  --disable_torch_compile \
  --ignore_layers "self_attn,pre_expert_proj,post_expert_proj,mlp.gate,vision_tower,vision_adapter,vision_projection,audio_tower,embed_tokens,lm_head,model.language_model.layers.0.mlp.experts,model.language_model.layers.61.mlp.experts" \
  --device_map auto \
  --format auto_round \
  --output_dir ./Muse-Spark-Open-816B-A42B-INT4

Notes:

• --ignore_layers uses substring matching against the full module name (if fp_layer in name). The two trailing entries model.language_model.layers.0.mlp.experts / ...layers.61.mlp.experts keep only the first and last layers' experts in BF16 (verified: they match exactly L0 & L61, 768 modules each, with no collateral on layers.10/20/... or the towers).
• Patterns that end in a digit get a . auto-appended by AutoRound (layers.6 → layers.6.), so prefer the explicit .mlp.experts suffix used above to avoid layers.6 matching layers.60.
• Keep the comma list clean — an empty item (e.g. an accidental double comma) makes AutoRound run name[-1] on an empty string and crash with IndexError during layer-config resolution.
• This is a weight-only build; unlike the MXFP4 / NVFP4 variants it does not quantize activations.

What is quantized (verified tensor-level)

Everything else — attention (self_attn.*), router (mlp.gate, FP32), pre/post_expert_proj, all norms & residual gates, the full vision tower (50 layers) and audio tower, the multimodal projectors, embed_tokens and lm_head — ships in full precision. In the -1008 byte breakdown the MoE experts account for ≈ 90% of weight bytes, so quantizing them drives essentially all of the compression while every accuracy-sensitive module (router, attention, multimodal perception) stays lossless.

Comparison with NVFP4

Compared against the NVIDIA reference build Muse-Spark-Open-816B-A42B-NVFP4 (modelopt, NVFP4, group_size=16, exclude_modules=["layers.0.","layers.61.","self_attn","lm_head"]), verified against its real weights:

Quantization scope is identical. The only difference is the numeric format:

Note: NVFP4's config.json does not list mlp.gate in exclude_modules, but modelopt does not quantize the MoE router by default — its real mlp.gate.weight tensors are float32 with no weight_scale, confirming the router is kept full-precision, same as this build.

Usage (vLLM)

Recommended inference parameters

Inherited from the base model (Muse-Spark-Open-816B-A42B) — quantization does not change them:

• Sampling: temperature = 1.0, top_p = 1.0, top_k disabled.
• Reasoning effort: set reasoning_effort to low / medium / high / xhigh (default xhigh); use high or xhigh for complex reasoning, coding, and agentic tasks, and low / medium for faster, shorter responses. Parsed by reasoning_parser=musespark1x.

Launch

Key launch parameters (grounded in the evaluation harness below):

• reasoning_parser=musespark1x — natively registered in vllm/reasoning/__init__.py.
• tokenizer_mode=auto — the checkpoint ships a standard PreTrainedTokenizerFast; do not pass the old musespark1x tokenizer mode (it is no longer registered and will error).
• dtype=bfloat16, config_format=auto, load_format=auto.
• Env: VLLM_WORKER_MULTIPROC_METHOD=spawn, FLASHINFER_DISABLE_VERSION_CHECK=1, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, TRANSFORMERS_OFFLINE=1.

Equivalent vllm serve (tensor-parallel size is deployment-dependent — ≈ 466 GB of weights needs ≥ 2× H200/B300 class GPUs):

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export FLASHINFER_DISABLE_VERSION_CHECK=1
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

vllm serve intel-ai/Muse-Spark-Open-816B-A42B-INT4 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --tokenizer-mode auto \
  --reasoning-parser musespark1x \
  --served-model-name Muse-Spark-Open-816B-A42B-INT4

Evaluation (lm-eval-harness · vLLM backend)

The following script is for the accuracy evaluation on piqa / mmlu / hellaswag and gsm8k:

export CUDA_VISIBLE_DEVICES=1,2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export FLASHINFER_DISABLE_VERSION_CHECK=1
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export TRANSFORMERS_OFFLINE=1

NUM_GPUS=2
GPU_MEMORY_UTILIZATION=${GPU_MEMORY_UTILIZATION:-0.9}

MODEL=${MODEL:-./Muse-Spark-Open-816B-A42B-INT4}
MODEL_ARGS="pretrained=${MODEL},tensor_parallel_size=${NUM_GPUS},max_model_len=8192,max_num_batched_tokens=4096,max_num_seqs=16,add_bos_token=True,gpu_memory_utilization=0.95,dtype=bfloat16,max_gen_toks=2048,enable_prefix_caching=False,reasoning_parser=musespark1x,tokenizer_mode=auto,config_format=auto,load_format=auto,enable_thinking=False"

# Multiple-choice / likelihood tasks
lm_eval \
  --model vllm \
  --model_args $MODEL_ARGS \
  --tasks piqa,mmlu,hellaswag \
  --batch_size 128 \
  --seed 42 \
  --output_path lm_eval_results

# Generative task (chat template + multi-turn few-shot)
lm_eval \
  --model vllm \
  --model_args $MODEL_ARGS \
  --tasks gsm8k \
  --batch_size 128 \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --seed 42 \
  --output_path lm_eval_results

Notes:

• tensor_parallel_size / CUDA_VISIBLE_DEVICES are set for a 2-GPU run; scale them to your host.

License & Attribution

Built with Muse Spark Open.

This model is a quantized derivative of Muse-Spark-Open-816B-A42B, released under the Muse Model License (license_name: muse-model-license; full text in LICENSE.md). Per the license, this notice, the license, and its accompanying Usage Policy must be retained in all copies and further derivatives, and products or services built on this model must prominently display "Built with Muse Spark Open".