Qwen3-30B-A3B ShapleyMCG K3/K4 validation reconstruction

This repository is the exact expanded BF16 validation model used for the reported WikiText KLD experiment. It installs the selected K3/K4 reconstructed expert matrices into the original Transformers checkpoint so the measured student logits can be independently reproduced.

Important format statement

The allocation has a logical average rate of 3.5 bits per MoE expert-weight element (9,216 K3 and 9,216 K4 matrix choices). This repository stores those reconstructions in BF16 and is therefore about the size of the BF16 base model. It is not a compact 3.5-bpw checkpoint and is not runtime-qualified. The current official BTX format couples gate/up bit rates, while this experiment selects gate and up independently; publishing it as official BTX would misrepresent the measured model.

Exact KLD result

Against sealed BF16 teacher logits on 2,047 next-token positions from the immutable WikiText-2-raw-v1 test-prefix window:

Allocation SHA256: 10bb3b71a2258500b281e3b37d57657e5e0ecf9318bb9c1a3fc6dbd2326e63c4
KLD report seal: f1ac3903116141faa18bf2f8253ebd875730451f8c0a23645b366fd5517a072b
Independent verification seal: b866e70b5e48433b3d9a014531acb0c460b3788198f3f8855830b6e58fd93fe6

Controlled causal-allocation result

The historical and causal arms use the same exact Qwen3-30B-A3B-Base revision, candidate reconstructions, 9,216 K3 plus 9,216 K4 matrix choices, sealed BF16 teacher, token IDs, SDPA attention implementation, and four-layer reanchoring cadence. Only the exact-rate expert-matrix allocation changes.

The causal allocation lowers KLD by 11.8963% and gains 1.0747 percentage points in top-1 agreement. It changes 9,392 of the 18,432 matrix choices without changing the exact logical rate or stored candidate payload budget. The independent float64 replay differs by at most 9.04e-13 per token.

Same-parent 20,480-position WikiText replication

The broader panel follows ExLlamaV3's published WikiText construction (ten consecutive 2,048-token rows), but both arms and the BF16 teacher are executed with Transformers SDPA. It is therefore a matched internal replication—not a claim that Transformers SDPA and ExLlamaV3-native attention are numerically identical.

The causal allocation lowers 20k-panel KLD by 7.7312% and gains 0.2686 percentage points in top-1 agreement. Independent float64 replay has maximum per-token differences of 0 for the causal arm and 0 for the historical arm. The complete teacher/student logits, per-token vectors, reports, manifests, and verification artifacts are retained in the reproducibility dataset.

The full candidates, calibration/Hessian artifacts, teacher and student logits, token window, manifests, hashes, and receipts are published in brandonmusic/shapleymcg-qwen3-30b-a3b-reproducibility. The executable pipeline and methodology are in brandonmmusic-max/shapleymcg.

Post-trained replication and complete result index

The same full causal allocator was subsequently run on the post-trained Qwen/Qwen3-30B-A3B parent using its separately fitted, retained MCG candidate inventory. On a matched 20,480-position eager WikiText panel at exact 3.5 routed-expert BPW, it lowered KLD from the predecessor routed-p2 allocation's 0.0411226321813 to 0.0403685517455 (1.833736%) and raised top-1 agreement from 0.924267578125 to 0.925683593750. Independent float64 replay had zero maximum tokenwise difference.

The allocation also transferred to TurboDerp's unchanged exact-3.5 expert candidate pool, lowering its matched KLD from 0.0302749179770 to 0.0292690766473 (3.322359%) and improving top-1 agreement by 0.273438 percentage points. The separately produced MCG pool measured 0.0457931025429 under the same allocation and body scope. That comparison does not isolate the codec: calibration corpus and hidden-state policy, rotation/scaling choices, procedural selection, and numeric encoder implementation also differ.

The stricter allocator-only test reconstructs both arms from the identical published TurboDerp K3/K4 candidates at exact 3.5 expert BPW. TurboDerp v0.0.1's carried-surplus rule measured KLD 0.0339415351804; full Aumann–Shapley/Fisher measured 0.0292690766473, a 13.766197% reduction, with a 0.600586 percentage-point top-1 gain and all 10 evaluation rows favorable. This proves an allocator-only advantage over the v0.0.1 rule on the common reconstructed candidate bytes; it is not a comparison against a native TurboDerp 3.5 checkpoint, which was not published for this model.

This model repository remains the Base-parent validation reconstruction; the post-trained logits and provenance are in brandonmusic/shapleymcg-qwen3-30b-a3b-posttrained-reproducibility. The GitHub complete results ledger and method specification distinguish parent, allocator generation, codec, dense-weight scope, evaluator, and token panel for every headline number.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "brandonmusic/Qwen3-30B-A3B-ShapleyMCG-K34-Validation-Reconstruction"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

Attribution

• Base model: the Qwen team, Qwen/Qwen3-30B-A3B-Base.
• EXL3/TRELLIS codec lineage: turboderp and ExLlamaV3 contributors.
• Aumann–Shapley quantization precedent: Joshua Hill, Saturation Makes Quantization Error Additive, and NVIDIA Model Optimizer PR #2183.
• Routed Fisher/Jacobian attribution, exact-rate reconciliation, and MCG integration are described and attributed in the GitHub methodology and sealed dataset card.
• Experiment, integration, and publication: Brandon Music / ShapleyMCG.