Muse-Spark-Open-816B-A42B TrellisMX K3 FP4

A 3-bit TrellisMX build of meta-models/Muse-Spark-Open-816B-A42B (BF16 base, revision 0fb02478b02565b2ce7fc57e6051c91172774ac9) whose routed experts decode to FP4 (E2M1) operands for block-scaled FP4 tensor-core math, with the image, video and audio towers kept. It is the FP4-operand sibling of Muse-Spark-Open-816B-A42B-TrellisMX-K3, which decodes the same 3-bit rate to E4M3 operands.

Built with Muse Spark Open. Use is subject to the Muse Model License (version: Muse Spark Open) and the Muse Spark Open Usage Policy (LICENSE.md, USE_POLICY.md).

Status: research build with a proof-of-concept serving image. The weights and the reference-path measurements below are complete. The serving image verdictai/muse-spark-trellismx:k3-fp4-tp4-20261017 (image ID sha256:5539082a51d25365ade5f552d98c7a9acc0302be7467bd939fc978e783ef824b) is the K3 serving image with this build's FP4 expert kernels added; in their single-GPU checks those kernels match the reference decode bit for bit. The full four-GPU serve of this checkpoint has not been run yet, so no speed, memory or KV-capacity figure is given here.

Run it (proof of concept)

You need four RTX PRO 6000 Blackwell 96 GB cards (SM120) with PCIe peer-to-peer between them, an NVIDIA driver with CUDA 13.4 support, the NVIDIA Container Toolkit, docker compose v2, and about 715 GB of disk (this checkpoint with its drafter 364 GB, its TP4 serving pack about 350 GB). Access to this repo is gated: request it on this page first.

1. Get the serving image and this checkpoint. The checkpoint includes drafter/, an MXFP8 copy of NVIDIA's DFlash2 drafter for this model:

docker pull verdictai/muse-spark-trellismx:k3-fp4-tp4-20261017
hf download brandonmusic/Muse-Spark-Open-816B-A42B-TrellisMX-K3-fp4 --local-dir /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3-FP4

2. Build the TP4 serving pack once (CPU only). It lays the checkpoint's expert files out per GPU without changing the weights, and checks every file against trellismx-manifest.json:

docker run --rm --network none --entrypoint python3 \
  -v /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3-FP4:/checkpoint:ro \
  -v /models/Muse-Spark-FP4-TP4-pack:/pack \
  verdictai/muse-spark-trellismx:k3-fp4-tp4-20261017 \
  -m muse_trellismx.pack_tp4 --checkpoint /checkpoint --out /pack

Adding --full-closure also decodes every expert on every rank to confirm the layout; that check takes far longer.

3. Serve from this repo's serving/ folder:

cd /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3-FP4/serving
export CHECKPOINT_ROOT=/models/Muse-Spark-Open-816B-A42B-TrellisMX-K3-FP4
export PACK_ROOT=/models/Muse-Spark-FP4-TP4-pack
export DRAFT_MODEL_ROOT=/models/Muse-Spark-Open-816B-A42B-TrellisMX-K3-FP4/drafter
bash serve.sh

The API is OpenAI-compatible on port 8010, model name muse-spark-trellismx-k3-fp4, with image, video and audio input through /v1/chat/completions. vLLM sizes the KV cache itself: the FP4 expert scales take more memory per GPU than the E4M3-operand build's, so that build's fixed KV setting does not apply here. If the first start runs out of memory, set GPU_MEMORY_UTILIZATION=0.95. MUSE_SPEC_DECODING=off serves without a drafter; NVIDIA's original BF16 DFlash2 card also works (download nvidia/Muse-Spark-Open-816B-A42B-DFlash2 and point DRAFT_MODEL_ROOT at it). serve.sh and docker-compose.yml list every setting; all of them can be overridden from the environment.

What is quantized

Fidelity

KL(BF16 teacher || this build) in nats, over the full vocabulary (the tokenizer's 201,818 ids; the 230 padded lm_head columns are dropped on both sides), in FP64, at every next-token row (row r predicts token r+1). Teacher: the BF16 logits in brandonmusic/Muse-Spark-Open-816B-A42B-BF16-Teacher-Logits, folder v2-2k/, revision 0bd19bd6. Intervals are window-block bootstrap (5,000 resamples, BCa) and Wilson for top-1.

Headline, 2,048-token windows (the context length used for the GLM-5.3-Flash TrellisMX measurements): evaluation set v2's 32 windows cut into 128 windows of 2,048 tokens, each run as a fresh sequence from position 0 (2,047 rows per window). 124 windows pass the 5% calibration-overlap screen (4 are left out); 122 of them contain text rows (two are entirely image placeholders). Text rows only, 218,850 rows:

Paired on the same rows, the serving formats add 0.0140 nats [0.0098, 0.0176] over weights only (ratio 1.59 [1.34, 1.78]). All 128 windows, without the overlap screen: 0.0381 [0.0324, 0.0452], top-1 94.20% (227,038 text rows). The 128 windows come from 32 source conversations; treating each conversation as one bootstrap block gives 0.0377 [0.0289, 0.0480].

Against the E4M3-operand build on the same 218,850 rows (its serving formats: 0.0330), this build adds 0.0047 nats [0.0034, 0.0060] (ratio 1.14 [1.11, 1.18]). Its weights alone are closer to the reference than that build's (0.0237 against 0.0269); the difference comes from the 4-bit activations.

By window type (text rows): text 0.0306, image 0.0389, video 0.0281, audio 0.1132. By text domain: agentic 0.0406, SWE agent 0.0791, math 0.0333, science 0.0267, competitive programming 0.0177, instruction following and chat 0.0176, multilingual 0.0175, long-context transduction 0.0042, safety 0.0246, web text 0.0295, legal 0.0287. By position in the window: 0-256 0.0693, 256-1,024 0.0381, 1,024-2,047 0.0296. By chat turn: assistant 0.0210 (top-1 95.84%), tool 0.0428, user 0.0651, system 0.0841, untemplated text 0.0287.

The 8,192-token measurement reported for the E4M3-operand build was not run for this build.

Not a fidelity measurement: the mean over all rows, including the 34,978 rows whose input is an image, video or audio placeholder, is 0.499 [0.325, 0.746], top-1 87.86% (253,828 rows). It is not one because the model makes no prediction at those rows: the processor fills those positions, and the BF16 teacher's top-1 equals the token that follows in 0 of them (70.0% on text rows), outside the [0.20, 0.995] band a row needs to count as a prediction. Their own KLD is 4.27 (image), 3.35 (video) and 0.58 (audio). What the model reads from the media reaches the text through attention, and that text is in the headline. Every row, placeholders included, is in evaluation/2k-windows/records/.

Checks: the teacher scored against itself is exactly 0 on every window; two independent runs of the serving-formats path give bit-identical per-token results on every window; on the text rows of every window with at least 32 text rows the teacher's top-1 agrees with the realized token inside the [0.20, 0.995] band; the scoring kernel matches the CPU reference on two 256-row slices of every window. The 2,048-token windows that share more than 5% of their content 13-grams (system turn and media placeholders excluded) with the data the build was calibrated on are left out of the headline.

How it was measured: the base model's decoder layers run in PyTorch, layer by layer on 4x NVIDIA B200, one window per forward, with this checkpoint's published files; the serving formats are reproduced bit for bit in that reference path (no native FP4 kernel path has been measured). Final norm and lm_head in FP32, as for the teacher. Everything in this section, per window, per domain, per position and per turn: evaluation/2k-windows/summary.json; per-token records: evaluation/2k-windows/records/.

Files

• model-0000N-of-00006.safetensors + model.safetensors.index.json: the non-expert tensors (MXFP8 text attention and every unquantized tensor).
• experts/layer-LLL-SSS-EEE.safetensors: the 3-bit routed experts, 32 experts per file: the trellis stream, the per-matrix FP32 scale, and the E4M3 block scales stored CSF-compressed (scale_csf, scale_csf_exceptions, scale_csf_starts; lossless).
• coupled/layer-LLL.safetensors: per-layer vectors the runtime applies with the experts.
• evaluation/2k-windows/: summary.json (every statistic above, per window, per domain, per position and per chat turn, for both paths), records/ (per-token records for both paths) and eval-windows.json (every evaluation window's source window, token-id SHA-256 and overlap screen).
• serving/: docker-compose.yml and serve.sh for the proof-of-concept serving image verdictai/muse-spark-trellismx:k3-fp4-tp4-20261017.
• drafter/: an MXFP8 copy of NVIDIA's DFlash2 drafter for this model (linear weights as E4M3 with a UE8M0 scale per 32 inputs); NVIDIA Open Model Agreement (see drafter/README.md).
• config.json (from the base release, unchanged) and hf_quant_config.json; tokenizer, processor and chat template files from the base release.
• trellismx-manifest.json: SHA-256 and size of every file, per-layer chunk lists and coupled-vector hashes.

Teacher logits

The BF16 teacher logits used for the KLD figures are in the dataset brandonmusic/Muse-Spark-Open-816B-A42B-BF16-Teacher-Logits (revision 0bd19bd6), folder v2-2k/.

Credits

• Base model. Muse Spark Open by Meta (meta-models/Muse-Spark-Open-816B-A42B), used under the Muse Model License and the Muse Spark Open Usage Policy (LICENSE.md, USE_POLICY.md).
• Serving image: Local Inference Lab and Luke Alonso. verdictai/muse-spark-trellismx:k3-fp4-tp4-20261017 is built on the vLLM image by Local Inference Lab and Luke Alonso (ghcr.io/local-inference-lab/vllm, https://github.com/local-inference-lab), a fork of vLLM (Apache-2.0, vLLM contributors). It also carries the open-source projects that image bundles, each under its own license: FlashInfer, LMCache, InstantTensor, xgrammar and fastokens.
• Kernels: b12x by Luke Alonso. The MoE, attention and GEMM kernels come from b12x (https://github.com/lukealonso/b12x, mirrored at https://github.com/local-inference-lab/b12x) by Luke Alonso, with Martin Vit and other contributors (Apache-2.0).
• CSF encoder: Local Inference Lab. The lossless compressed scale format (CSF) the expert scales are stored in, and its encoder and decoder, are Local Inference Lab's work, shipped in Local Inference Lab's FlashInfer.
• TrellisMX: QTIP and EXL3. TrellisMX, Brandon M. Music's trellis-coded format, was inspired by QTIP (Albert Tseng, Qingyao Sun, David Hou and Christopher De Sa, "QTIP: Quantization with Trellises and Incoherence Processing", NeurIPS 2024) and by ExLlamaV3 and its EXL3 format by turboderp and contributors (https://github.com/turboderp-org/exllamav3, MIT License).
• Codec changes: QSRT and Luke Alonso. Decoding the trellis directly into FP4 (E2M1) tensor-core operands was inspired by Luke Alonso's KQuant/QSRT work and the b12x W4A8 trellis path, which decodes the QSRT trellis to E4M3.
• Methods. The encoder's Hessian-based error feedback follows GPTQ (Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, ICLR 2023). The incoherence processing and rotations follow QuIP (Jerry Chee, Yaohui Cai, Volodymyr Kuleshov and Christopher De Sa, NeurIPS 2023), QuIP# (Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov and Christopher De Sa, ICML 2024), QuaRot (Saleh Ashkboos et al., NeurIPS 2024) and SpinQuant (Zechun Liu et al., 2024). The UE8M0 block scales follow the OCP Microscaling (MX) Formats Specification (Bita Darvish Rouhani et al., 2023).
• Evaluation data. The evaluation windows behind the KLD figures (and the teacher logits) were built from public datasets, each under the license on its own dataset card: NVIDIA's Nemotron-SFT-Agentic-v2, Nemotron-SFT-Math-v4, Nemotron-SFT-Science-v2 (CC-BY-SA-4.0), Nemotron-SFT-Competitive-Programming-v2, Nemotron-SFT-Instruction-Following-Chat-v3, Nemotron-SFT-Multilingual-v2, Nemotron-SFT-Long-Context-Transduction-v1 (OpenMDW-1.1), Nemotron-SFT-Safety-v2, Nemotron-Pretraining-Legal-v1 (CC-BY-4.0), Nemotron-VLM-Dataset-v2 (CC-BY-4.0) and Nemotron-Pretraining-Dataset-sample (Nemotron-CC text; NVIDIA Open Data License Agreement); Nebius's SWE-agent-trajectories (CC-BY-4.0); LibriSpeech (openslr/librispeech_asr; Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur, ICASSP 2015; CC-BY-4.0); and FLEURS (google/fleurs; Google; CC-BY-4.0). Each window's sources are listed in evaluation/2k-windows/eval-windows.json.
• Drafter: NVIDIA. drafter/ is NVIDIA's DFlash2 drafter for this model, converted to MXFP8, under the NVIDIA Open Model Agreement (drafter/README.md).