Muse-Spark-Open-816B-A42B TrellisMX K3

A 3-bit TrellisMX build of meta-models/Muse-Spark-Open-816B-A42B (BF16 base, revision 0fb02478b02565b2ce7fc57e6051c91172774ac9) sized for four 96 GB RTX PRO 6000 Blackwell cards at tensor parallel 4, with the image, video and audio towers kept.

Built with Muse Spark Open. Use is subject to the Muse Model License (version: Muse Spark Open) and the Muse Spark Open Usage Policy (LICENSE.md, USE_POLICY.md).

Status: research build. The weights and the reference-path measurements below are complete. The serving image verdictai/muse-spark-trellismx:k3-fp8-tp4-20261017 runs this checkpoint on four RTX PRO 6000 Blackwell 96 GB cards at tensor parallel 4, with DFlash2 speculative decoding, full CUDA graphs and image, video and audio input; serving/ holds its docker-compose.yml, serve.sh and the measured serving figures (serving/SERVING.md).

Run it

You need four RTX PRO 6000 Blackwell 96 GB cards (SM120) with PCIe peer-to-peer between them, an NVIDIA driver with CUDA 13.4 support, the NVIDIA Container Toolkit, docker compose v2, and about 650 GB of disk (this checkpoint with its drafter 342 GB, its TP4 serving pack 303 GB). Access to this repo is gated: request it on this page first.

1. Get the serving image and this checkpoint. The checkpoint includes drafter/, an MXFP8 copy of NVIDIA's DFlash2 drafter for this model, used for the figures below:

docker pull verdictai/muse-spark-trellismx:k3-fp8-tp4-20261017
hf download brandonmusic/Muse-Spark-Open-816B-A42B-TrellisMX-K3 --local-dir /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3

2. Build the TP4 serving pack once (CPU only). It lays the checkpoint's expert files out per GPU without changing the weights, and checks every file against trellismx-manifest.json:

docker run --rm --network none --entrypoint python3 \
  -v /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3:/checkpoint:ro \
  -v /models/Muse-Spark-TP4-pack:/pack \
  verdictai/muse-spark-trellismx:k3-fp8-tp4-20261017 \
  -m muse_trellismx.pack_tp4 --checkpoint /checkpoint --out /pack

Adding --full-closure also decodes every expert on every rank to confirm the layout; that check takes far longer.

3. Serve from this repo's serving/ folder (the first start compiles kernels, about 6.5 minutes from empty caches):

cd /models/Muse-Spark-Open-816B-A42B-TrellisMX-K3/serving
export CHECKPOINT_ROOT=/models/Muse-Spark-Open-816B-A42B-TrellisMX-K3
export PACK_ROOT=/models/Muse-Spark-TP4-pack
export DRAFT_MODEL_ROOT=/models/Muse-Spark-Open-816B-A42B-TrellisMX-K3/drafter
bash serve.sh

The API is OpenAI-compatible on port 8010, model name muse-spark-trellismx-k3, with image, video and audio input through /v1/chat/completions:

curl -s http://127.0.0.1:8010/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"muse-spark-trellismx-k3","messages":[{"role":"user","content":"Hello"}],"max_tokens":256}'

Measured on four RTX PRO 6000 Blackwell cards with the included drafter: 1,040,217 KV tokens; decode 63.7 tok/s for one request and 357.6 tok/s aggregate at 16 requests (61.1 and 356.0 at 16K context). NVIDIA's original BF16 DFlash2 card also works (download nvidia/Muse-Spark-Open-816B-A42B-DFlash2 and point DRAFT_MODEL_ROOT at it); it takes 1.33 GiB more memory per GPU, so serve.sh lowers the KV cache to 921,335 tokens (decode 63.1 and 351.1 tok/s). MUSE_SPEC_DECODING=off serves without a drafter. serving/SERVING.md has every setting and its default, the image details and the full measurements.

What is quantized

Fidelity

KL(BF16 teacher || this build) in nats, over the full vocabulary (the tokenizer's 201,818 ids; the 230 padded lm_head columns are dropped on both sides), in FP64, at every next-token row (row r predicts token r+1). Teacher: the BF16 logits in brandonmusic/Muse-Spark-Open-816B-A42B-BF16-Teacher-Logits, revision 0bd19bd6, folders v2-2k/ (2,048-token windows) and v2/ (8,192-token windows). Intervals are window-block bootstrap (5,000 resamples, BCa) and Wilson for top-1.

Headline, 2,048-token windows (the context length used for the GLM-5.3-Flash TrellisMX measurements): evaluation set v2's 32 windows cut into 128 windows of 2,048 tokens, each run as a fresh sequence from position 0 (2,047 rows per window). 124 windows pass the 5% calibration-overlap screen (4 are left out); 122 of them contain text rows (two are entirely image placeholders). Text rows only, 218,850 rows:

Paired on the same rows, the serving formats add 0.0061 nats [0.0000, 0.0105] over weights only (ratio 1.23 [1.00, 1.47]). All 128 windows, without the overlap screen: 0.0333 [0.0283, 0.0399], top-1 94.52% (227,038 text rows). The 128 windows come from 32 source conversations; treating each conversation as one bootstrap block gives 0.0330 [0.0251, 0.0426].

By window type (text rows, 2,048-token windows): text 0.0259, image 0.0362, video 0.0254, audio 0.1038. By text domain: agentic 0.0353, SWE agent 0.0675, math 0.0277, science 0.0218, competitive programming 0.0150, instruction following and chat 0.0150, multilingual 0.0145, long-context transduction 0.0033, safety 0.0210, web text 0.0256, legal 0.0243. By position in the window: 0-256 0.0590, 256-1,024 0.0341, 1,024-2,047 0.0258. By chat turn: assistant 0.0201 (top-1 95.81%), tool 0.0381, user 0.0583, system 0.0687, untemplated text 0.0251.

An earlier version of this card gave 0.0386 for 2,048-token context, computed on the first 2,047 rows of each 8,192-token window. Those rows all begin at the system prompt, the highest-KLD position, so that figure did not represent 2,048-token windows; the measurement above replaces it.

Full 8,192-token windows (the 32 windows of evaluation set v2, 8,191 rows each, teacher folder v2/; KLD is lower at this length because most rows have long context: 0.063 in the first 256 positions, 0.015 past position 4,096), text rows, 227,119 rows:

Paired on the same 8,192-token rows, the serving formats add 0.0057 nats [0.0049, 0.0067] over weights only (ratio 1.39 [1.34, 1.46]).

By window type (serving formats, text rows, 8,192-token windows):

By text domain: agentic 0.0255, SWE agent 0.0418, math 0.0143, science 0.0136, competitive programming 0.0108, instruction following and chat 0.0102, multilingual 0.0092, long-context transduction 0.0030, safety 0.0139, web text 0.0257, legal 0.0218. By position in the window: 0-256 0.0626, 256-1K 0.0451, 1K-4K 0.0195, 4K-8K 0.0148. By chat turn: assistant 0.0092 (top-1 97.03%), tool 0.0215, user 0.0344, system 0.0385, untemplated text 0.0259.

Not a fidelity measurement: the mean over all rows, including the rows whose input is an image, video or audio placeholder, is 0.469 [0.302, 0.708] at 2,048 tokens (253,828 rows, top-1 88.40%) and 0.4536 [0.2141, 0.8215] at 8,192 tokens (262,112 rows, top-1 90.49%). It is not one because the model makes no prediction at those rows: the processor fills those positions, and the BF16 teacher's top-1 equals the token that follows in 0 of the 34,978 (2,048) and 34,993 (8,192) placeholder rows (70.0% and 75.0% on text rows), outside the [0.20, 0.995] band a row needs to count as a prediction. Their own KLD at 2,048 tokens is 4.05 (image), 3.08 (video) and 0.51 (audio). What the model reads from the media reaches the text through attention, and that text is in the headline. Every row, placeholders included, is in evaluation/2k-windows/records/ and evaluation/records/.

Checks: the teacher scored against itself is exactly 0 on every window; two independent runs of the serving-formats path give bit-identical per-token results on every window at both lengths; on the text rows of every window with at least 32 text rows the teacher's top-1 agrees with the realized token inside the [0.20, 0.995] band (the band is evaluated on text rows because placeholder rows are not predictions); the teacher recomputed with different batch partners differs from itself by 0.00009 nats on text rows; the scoring kernel matches the CPU reference to 3e-13. The 8,192-token evaluation windows share at most 5% of their content 13-grams (system turn and media placeholders excluded) with the data the build was calibrated on, repeat no conversation and share no image, video or audio item with it; cut into 2,048-token windows, four pieces exceed 5% and are left out of the 2,048-token headline.

How it was measured: the base model's decoder layers run in PyTorch, layer by layer on 4x NVIDIA B200, one window per forward, with this checkpoint's published files; the serving formats are reproduced bit for bit in that reference path (the native SM120 kernel path has not been measured). Final norm and lm_head in FP32, as for the teacher. Everything in this section, per window, per domain, per position and per turn: evaluation/2k-windows/summary.json (2,048 tokens) and evaluation/summary.json (8,192 tokens); per-token records: evaluation/2k-windows/records/ and evaluation/records/.

Files

• model-0000N-of-00006.safetensors + model.safetensors.index.json: the non-expert tensors (MXFP8 text attention and every unquantized tensor).
• experts/layer-LLL-SSS-EEE.safetensors: the 3-bit routed experts, 32 experts per file, with their UE8M0 scales stored CSF-compressed (scale_csf, scale_csf_exceptions; lossless).
• coupled/layer-LLL.safetensors: per-layer vectors the runtime applies with the experts.
• evaluation/: summary.json (every 8,192-token statistic above, per window, per domain, per position and per chat turn, for both paths), records/ (per-token records for both paths) and eval-windows.json (every evaluation window's sources, token-id SHA-256 and screening result); evaluation/2k-windows/: the same for the 2,048-token windows (summary.json, records/, eval-windows.json with each window's source window and overlap screen).
• serving/: SERVING.md (setup and measured figures), docker-compose.yml and serve.sh for the serving image verdictai/muse-spark-trellismx:k3-fp8-tp4-20261017.
• drafter/: an MXFP8 copy of NVIDIA's DFlash2 drafter for this model (linear weights as E4M3 with a UE8M0 scale per 32 inputs), used for the serving figures; NVIDIA Open Model Agreement (see drafter/README.md).
• config.json (from the base release, unchanged) and hf_quant_config.json; tokenizer, processor and chat template files from the base release.
• trellismx-manifest.json: SHA-256 and size of every file, per-layer chunk lists and coupled-vector hashes.

Teacher logits

The BF16 teacher logits used for the KLD figures are in the dataset brandonmusic/Muse-Spark-Open-816B-A42B-BF16-Teacher-Logits (revision 0bd19bd6): folder v2-2k/ for the 2,048-token windows and v2/ for the 8,192-token windows.

Credits

• Base model. Muse Spark Open by Meta (meta-models/Muse-Spark-Open-816B-A42B), used under the Muse Model License and the Muse Spark Open Usage Policy (LICENSE.md, USE_POLICY.md).
• Serving image: Local Inference Lab and Luke Alonso. verdictai/muse-spark-trellismx:k3-fp8-tp4-20261017 is built on the vLLM image by Local Inference Lab and Luke Alonso (ghcr.io/local-inference-lab/vllm, https://github.com/local-inference-lab), a fork of vLLM (Apache-2.0, vLLM contributors). It also carries the open-source projects that image bundles, each under its own license: FlashInfer, LMCache, InstantTensor, xgrammar and fastokens.
• Kernels: b12x by Luke Alonso. The MoE, attention and GEMM kernels come from b12x (https://github.com/lukealonso/b12x, mirrored at https://github.com/local-inference-lab/b12x) by Luke Alonso, with Martin Vit and other contributors (Apache-2.0).
• CSF encoder: Local Inference Lab. The lossless compressed scale format (CSF) the expert scales are stored in, and its encoder and decoder, are Local Inference Lab's work, shipped in Local Inference Lab's FlashInfer.
• TrellisMX: QTIP and EXL3. TrellisMX, Brandon M. Music's trellis-coded format, was inspired by QTIP (Albert Tseng, Qingyao Sun, David Hou and Christopher De Sa, "QTIP: Quantization with Trellises and Incoherence Processing", NeurIPS 2024) and by ExLlamaV3 and its EXL3 format by turboderp and contributors (https://github.com/turboderp-org/exllamav3, MIT License).
• Codec changes: QSRT and Luke Alonso. Decoding the trellis directly into FP8 (E4M3) tensor-core operands was inspired by Luke Alonso's KQuant/QSRT work and the b12x W4A8 trellis path, which decodes the QSRT trellis to E4M3.
• Methods. The encoder's Hessian-based error feedback follows GPTQ (Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, ICLR 2023). The incoherence processing and rotations follow QuIP (Jerry Chee, Yaohui Cai, Volodymyr Kuleshov and Christopher De Sa, NeurIPS 2023), QuIP# (Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov and Christopher De Sa, ICML 2024), QuaRot (Saleh Ashkboos et al., NeurIPS 2024) and SpinQuant (Zechun Liu et al., 2024). The UE8M0 block scales follow the OCP Microscaling (MX) Formats Specification (Bita Darvish Rouhani et al., 2023).
• Evaluation data. The evaluation windows behind the KLD figures (and the teacher logits) were built from public datasets, each under the license on its own dataset card: NVIDIA's Nemotron-SFT-Agentic-v2, Nemotron-SFT-Math-v4, Nemotron-SFT-Science-v2 (CC-BY-SA-4.0), Nemotron-SFT-Competitive-Programming-v2, Nemotron-SFT-Instruction-Following-Chat-v3, Nemotron-SFT-Multilingual-v2, Nemotron-SFT-Long-Context-Transduction-v1 (OpenMDW-1.1), Nemotron-SFT-Safety-v2, Nemotron-Pretraining-Legal-v1 (CC-BY-4.0), Nemotron-VLM-Dataset-v2 (CC-BY-4.0) and Nemotron-Pretraining-Dataset-sample (Nemotron-CC text; NVIDIA Open Data License Agreement); Nebius's SWE-agent-trajectories (CC-BY-4.0); LibriSpeech (openslr/librispeech_asr; Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur, ICASSP 2015; CC-BY-4.0); and FLEURS (google/fleurs; Google; CC-BY-4.0). Each window's sources are listed in evaluation/eval-windows.json.
• Drafter: NVIDIA. drafter/ is NVIDIA's DFlash2 drafter for this model, converted to MXFP8, under the NVIDIA Open Model Agreement (drafter/README.md).