Muse Spark Open: last-4-layer partial

Built with Muse Spark Open.

A 4-layer slice of Muse Spark Open for testing training and serving pipelines on a single 8-GPU node. It is not a usable language model: its generations are meaningless.

Contents

• Decoder layers 58–61 of the 62-layer language model, renumbered 0–3. Taking the last layers keeps the attention schedule, which counts from the top: layers 0–2 use sliding-window attention and layer 3 uses full attention without positional encoding.
• text_config.num_hidden_layers is 4, and the per-layer lists layer_types, layer_rope_theta and mlp_layer_types keep only those layers.
• Every other tensor is unchanged: embeddings, final norm, LM head, and the vision and audio towers.
• Tokenizer, chat template, processor and generation configs are the base model's files, unchanged.
• 12 safetensors shards, 108 GiB. Tensors are BF16, except the MoE router weights and biases, which stay FP32 as in the base model.

Provenance

Extracted from meta-models/Muse-Spark-Open-816B-A42B at revision e80856c8 with scripts/muse_spark/extract_partial_checkpoint.py from the muse-spark-open branch of Miles. Revision 0fb02478 of the base model has the same weight shards and configs and changes only chat_template.jinja among the files used here; this repository ships the 0fb02478 chat template.

Use with Miles

On one 8 × H200 node, from a checkout of the Miles muse-spark-open branch, a 3-rollout smoke test:

ray start --head --port 6379 --num-gpus 8
MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_muse_spark_64xh200.py \
  --model-org RadixArk --model-name Muse-Spark-Open-last4-partial \
  --megatron-model-type muse-spark-last4-partial \
  --num-nodes 1 \
  --tensor-model-parallel-size 4 --pipeline-model-parallel-size 2 \
  --expert-model-parallel-size 4 --decoder-first-last-pipeline-num-layers 2 \
  --rollout-num-gpus-per-engine 8 \
  --num-rollout 3 --rollout-batch-size 4 --n-samples-per-prompt 4

License

Distributed under the license of the base model, the Muse Model License; see LICENSE.md and USE_POLICY.md.