Muse Spark Open 816B A42B — EXL3 mixed precision

Built with Muse Spark Open

A calibrated EXL3 conversion of the original BF16 checkpoint. Experts use 4 bits per weight in decoder layers 0–60 and 6 bits in the final decoder layer, 61. Attention and expert bridge projections use 6 bits. The embedding and output head remain BF16. Routers, correction biases, normalization weights, residual gates, and all vision/audio weights retain their original precision.

Source: meta-models/Muse-Spark-Open-816B-A42B, revision 0fb02478b02565b2ce7fc57e6051c91172774ac9. Every original tensor is accounted for in the inventory; no modality weights were omitted. Retained tensors were checked for bitwise equality against the original checkpoint.

Quantization method

Conversion uses ExLlamaV3 at 058b8630fd081f3ff46fce6246955b157c3a2e3b through quantize_exl3_batch, with the mul1 codebook, block Hadamard transforms, calibrated LDLQ error feedback, per-channel FP16 scales, output scaling enabled, and the upstream post-quantization scale refit. Hessians accumulate FP32 products of the observed BF16 module inputs. Diagonal regularization is 0.025 times the mean Hessian diagonal. Seeds are derived deterministically from seed 42 and each full tensor name.

Conversion streams one decoder layer at a time. Calibration inputs for each subsequent layer are propagated through saved quantized weights reconstructed to FP16 and rounded to BF16. Within a layer, the quantization Hessians are collected with that layer’s original weights. Expert gate and up projections share their input Hessian; down projections have separate Hessians. The original routing, residual gates, normalization order and BF16 rounding are preserved. In particular, the post-expert norm is applied before routing-weight multiplication and expert summation.

Calibration uses 524,288 tokens in 256 sequences of 2,048 tokens, equally divided among WikiText-103 train, The Stack small code sample, GSM8K train, and UltraChat train conversations. This is primarily English text/code calibration. Dataset revisions, row identifiers, deduplication rules, tokenizer/template hashes and token hashes are in data-manifest.json and selected-documents.jsonl. Raw dataset text and token arrays are not redistributed.

Across the 62 layers, 28 expert instances received zero naturally routed tokens. Those experts received supplemental counterfactual calibration: routing was bypassed to feed real post-bridge latent inputs sampled evenly from every calibration batch, up to 16,384 inputs per expert. This improves calibration coverage but is not evidence of natural routing or multimodal coverage. Natural and supplemental counts are reported separately in each layer’s coverage file. No uncalibrated quantizer fallback was used in this release.

Measured comparison with original BF16

The baseline loads the pinned original BF16 checkpoint using the recorded Transformers reference implementation. Both models use identical inputs, positions, masks, tokenization and scoring. These are our paired measurements, not publisher-reported benchmark scores. The evaluation set is separate from calibration and contains 124867 input tokens. Exact normalized-text duplicate checks and a five-word SimHash filter were applied; semantic overlap cannot be ruled out.

Perplexity sequences contain 2,048 tokens, except the separate 8,192-token WikiText sequence. They use packed documents with the token boundaries recorded in the supplied scripts and score all next-token positions, including questions and answers in the math split. These scores are not interchangeable with other WikiText or benchmark preprocessing protocols.

ARC and MMLU use zero-shot raw question/choice prompts and candidate-letter likelihoods. HellaSwag uses continuation likelihood with both sum and token-normalized scoring. The fixed subsets and custom protocols are fully recorded. No chain-of-thought generation, executable coding pass@1, tool-use success rate, or multimodal accuracy was measured. Confidence intervals use 10,000 paired bootstrap resamples of questions, seed 42. Zero-width paired intervals can occur when both models have the same per-question correctness indicators on a subset; they do not establish identical performance on unseen data.

These distribution comparisons are token-weighted over all evaluation input positions, including the repeated shared prefixes of HellaSwag candidate continuations. Top-1 disagreement measures deviation from the BF16 model, not task error. dequantized_bf16 isolates stored-weight distortion plus BF16 reconstruction rounding; packed also includes the EXL3 adapter’s FP16 linear arithmetic. Additional INT8-cache KL(packed → packed+INT8) averages 0.00496069 nats.

NMSE, Frobenius error, cosine similarity, SQNR and maximum absolute error compare the original matrices with reconstructions from saved and reloaded packed tensors. The Hessian proxy is the upstream quantizer’s reported error after FP32 scale refitting, before scales are rounded to FP16 for storage; it is not recomputed from the reloaded tensors. Its aggregate is a parameter-weighted mean of per-matrix normalized errors, not a global output-error normalization. The summary and complete evaluation include further quantiles, NLL, all four execution modes, per-domain divergences, top-10 retention and confidence intervals. The metrics/ directory contains every matrix’s errors, calibrated proxy, packed-kernel checks, per-layer hidden-state error and route agreement, cache reconstruction errors, coverage and per-example scores.

Per-layer hidden-state and routing diagnostics use the first 8,192 WikiText evaluation tokens. Aggregate INT8 roundtrip errors below compare each quantized model K/V tensor before and after cache encoding, across valid evaluation tokens:

The automated release criteria and outcomes are in release-gates.json.

Four-node DGX Spark memory plan

Actual tensor payload: 398.556 GiB; safetensors files including headers: 398.578 GiB. This is a mixed-precision release; “4-bit” describes most expert matrices, with the final expert layer at 6 bits.

Context capacity sensitivity, keeping the same all-weights placement, 6 GiB runtime allowance and 4 GiB reserve:

The cache budget is 262,144 aggregate tokens for one full-context request, including prompt and generated tokens, with 16 full-attention layers and 46 sliding-window layers capped at 2,048 cached positions each. K and V each use one INT8 value per element and an independent FP32 scale per token per KV head, with four of the model’s 16 KV heads on each rank. A further 1% cache allocation margin is included. Multiple simultaneous requests need additional sliding-window storage. Prefix caching has no allocation in this plan.

Experts, attention, embedding and head weights are sharded four ways; expert bridge projections, routers and norms are replicated. All EXL3 scale vectors and markers are conservatively counted as replicated. The all-weights plan additionally requires distributed vision/audio matrix and convolution weights as listed in multimodal-placement.json. Replicating every frontend on every node does not satisfy this conservative budget. Loading must stream each rank’s weights or use pre-sharded files; duplicate full CPU/GPU weight copies are not budgeted.

This is a capacity model based on actual tensor sizes, not a measured DGX deployment. The runtime allowance includes CUDA contexts/graphs, communication, pinned staging, and chunked prefill/decode scratch; its sufficiency is not yet validated. DGPP engine/kernel support remains necessary. See the complete plan.

The longest evaluated context is 8,192 tokens. INT8 K/V was tested by quantize/dequantize simulation after K normalization/position encoding, using symmetric max-absolute scaling, round-to-nearest and the range [-127,127]. This does not validate a serving cache allocator or 262K-context accuracy. FP8 cache arithmetic was not measured; only its equivalent one-byte element storage is covered by the capacity calculation.

Other requested quantizations: capacity only

These pinned alternatives were inspected through every safetensors header. Payload sizes were reconciled against file sizes; nominal bit widths alone were not used. Totals below use the same 262,144-token INT8 cache, 6 GiB workspace allowance and 4 GiB reserve per node.

NVFP4 and Intel INT4 exceed the 115 GiB-per-node plan even with fully distributed matrix weights. MLX 3-bit fits this capacity plan only when its BF16 expert bridges are distributed too; its inspected files contain vision weights but no audio weights. The EXL3 plan above retains all modalities and conservatively replicates its 6-bit bridges and all EXL3 scale vectors. No engines or accuracy scores for the alternative formats were tested. See the pinned measurements and placement assumptions.

Loading and reproducibility

Native ExLlamaV3 architecture support for Muse is not supplied by this release. The included Transformers adapter patch preserves Muse’s post-expert normalization. Packed execution was evaluated on a single RTX PRO 6000 Blackwell Server Edition by streaming layers; a complete model cannot reside on that one GPU.

Use the pinned ExLlamaV3 revision and apply exllamav3-muse-norm.patch, then register the adapter before loading:

from exllamav3.integration.transformers import register
register()
# Load using a supported placement with sufficient memory.
# This registration alone does not implement multi-node DGPP execution.

The recipe, software environment, source hashes, packed-file hashes, and conversion/evaluation scripts make the process reviewable. To repeat conversion, place the scripts and recipe in a scratch workspace with source/, data/, work/, output/, and the pinned patched exllamav3/ checkout. Copy the patch and recipe into that workspace too. Install the recorded software versions and apply the patch before running the checks below. Use your own authorized Hugging Face credentials through the standard token store. The source BF16 checkpoint requires approximately 1.64 TB of disk; the full conversion also needs substantial output and scratch space.

Run from the scratch workspace after placing the scripts directly in it:

python download_source.py
python verify_source.py
python prepare_data.py
python test_reference_path.py
python test_streaming.py
python test_evaluation.py
python test_hf_loading.py
python run_quantization.py --stop-after-layer 62
python evaluate_head.py
python inspect_alternatives.py
python assemble_release.py
python make_card.py
python review_release.py

No direct paired comparison against NVFP4, Intel INT4, or MLX 3-bit was performed. This card does not claim superiority over those quantizations. Multilingual and multimodal quality, generation-based reasoning, and long-context retrieval require further evaluation.

License

The original Muse Model License, copyright notice, and Usage Policy apply. Dataset revisions and available license metadata are recorded in the data manifest; raw calibration/evaluation datasets are not included.