GLM-5.2-EXL3-DENSE6-MTP78

A mixed-precision EXL3 serving checkpoint of GLM-5.2 for 4× RTX PRO 6000 Blackwell (sm_120a): rank-sliced TR3 3.0 bpw routed experts plus EXL3 6.0 bpw dense path (375 tensors), MTP speculative-decode head retained. 279.68 GiB on disk (15 GiB smaller than the 3.0 bpw + BF16-dense parent), +43.7 % KV-cache capacity, quality at the parent's noise floor.

Serving configs, patches and full measurement record: https://github.com/brandonmmusic-max/GLM-5.2-EXL3-DENSE6-MTP78 — mirrored here under serving/. That repo carries the serving configs, the two required patches, and every measurement (JSON + logs) behind the numbers below.

Quant recipe

• Base: GLM-5.2, via the 3.0-bpw TR3 parent and the MTP-78 lineage recorded in the checkpoint configuration as malaiwah/GLM-5.2-EXL3-TR3-MTP78/3bpw-keep0.
• Routed experts: TR3 3.0 bpw rank-sliced (hybrid_tr3_tail.moe_layers=[3,78]), byte-copied from the parent checkpoint.
• Dense path: 375 tensors moved BF16 → EXL3 6.0 bpw — 75 each of o_proj, q_b_proj, shared_experts.{gate,up,down}_proj (layers 3–77). Per-tensor encode NMSE 3.8–5.1e-04.
• Held in BF16 deliberately: lm_head, embed_tokens, router mlp.gate, DSA indexer, kv_b_proj, q_a_proj+kv_a_proj_with_mqa (packed fused_qkv_a_proj, n=576 fails Hadamard-128), all 15 dense tensors of layers 0–2, and the non-routed components of MTP layer 78. Layer-78 routed experts remain inherited EXL3 weights; the whole MTP layer is not BF16.
• Why 6.0 bpw and not FP8/MXFP8 for the dense path: measured on real production weights, EXL3-6's output error floor (~2.0e-2 rel. Frobenius) is below fp8 per-out-channel (2.64e-2) and MXFP8 block=32 (2.39e-2) — e4m3's 3 mantissa bits are the binding constraint, not scale granularity. An fp8-dense experiment measured KLD 9.505 (vs 0.114) and was rejected.

Serving stack

Reference image (recommended): voipmonitor/vllm:gilded-gnosis-v20-vllm69ba80b-sia2ea608-fi801d57a-cu132-20260730-r13 (registry digest sha256:02796036c96a52fda0919aa260c45c70bc97d8e662a6ae5e614b5f987c20851b). Also validated on the 20260729 r11 image (…-vllm9502cc7-side7739a-…-r11).

Stock images do NOT serve this model as-is — two small patches are required (see patches/, applied via Dockerfile.r13pf):

1. patch_exl3_num_active.py — required or every TP worker dies at load with TypeError: exl3_moe(): incompatible function arguments. The vendored exllamav3 extension takes 30 args (num_active appended); vLLM's call site passes 29. Appends -1 (0 would silently zero the routed-expert output). Present in r11, r12 and r13 unchanged; until it lands upstream, this needs either the patch or a ~1-line vLLM PR. The checkpoint format itself (mixed 3.0/6.0 bpw, rank-sliced hybrid tail) loads natively — the loader resolves codebooks per layer and Exl3LinearMethod never references a global bit width — so no "new model type" PR is needed; only this call-site arity fix.
2. patch_exl3_prefetch.py — restores exllamav3's >144-row reconstruct dispatch for the dense path (materialize once + one cuBLAS HGEMM instead of re-decoding the trellis per 16-row m-tile). Without it, dense-quantized prefill loses ~25–37 %. Measured 4.76× on o_proj at M=2048; numerically verified vs the fused path (≤3.5e-03 worst rel. deviation). The file also contains an env-gated (default-off, falsified-on-measurement) aux-stream prefetch experiment — leave VLLM_EXL3_PREFETCH=0.

Build: cd patches && docker build -f Dockerfile.r13pf -t glm52-r13-pf:v1 . Then ./serve.sh (edit volume paths in the compose first).

Config and env as benchmarked (verified from /proc/<pid>/environ)

KV_CACHE_DTYPE=nvfp4_ds_mla   KV_FP8_ROPE=1   VLLM_NVFP4_MLA_DYNAMIC_SCALE=1
VLLM_NVFP4_MLA_SCALES_FILE=   (explicitly empty → dynamic scale selection)
MOE_BACKEND=b12x  MOE_MODE=a16  DCP=4  DCP_KV_CACHE_INTERLEAVE_SIZE=1
VLLM_EXL3_RECONSTRUCT_M=144   VLLM_EXL3_PREFETCH=0
--attention-backend B12X_MLA_SPARSE   --quantization exl3
--max-num-seqs 8   --max-num-batched-tokens 3072
cudagraph_capture_sizes=[4..32]   MTP-3 speculative decoding
GPU_MEMORY_UTILIZATION=0.95 (launcher default)

Traps written in blood:

• nvfp4_ds_mla survives only under B12X_MLA_SPARSE; anything else silently coerces to fp8_ds_mla. Dynamic NVFP4 scaling requires KV_FP8_ROPE=1.
• The launcher chain defaults KV_CACHE_DTYPE to fp8 — pin it as a literal in the compose and verify from /proc/<pid>/environ, never from docker inspect (the launcher hard-exports values that don't appear there).
• The first boot on a fresh compile cache JIT-compiles kernels during serving (r13 resolves its fused-MoE launch per forward with a live-batch cache key). Mount a persistent cache volume and treat the first session as warmup — never benchmark it. (One cold arm in our campaign read −25 % prefill purely from this.)

Hardware

4× NVIDIA RTX PRO 6000 Blackwell Workstation (2× Max-Q at a 300 W hard cap), sm_120a, 188 SMs, 96 GB GDDR7 each, PCIe Gen5, no NVLink. TP4 + DCP4.

KV-cache capacity (as served, GMU 0.95)

Quality

KLD harness: 2048-token teacher-forced eval vs BF16 reference logits, n=5 runs, TP4 eager. Reference arms on the parent TR3 checkpoint: uncalibrated 0.151127 / static scales 0.116195 / dynamic 0.112645 / fp8-KV 0.100246.

Speed (position-matched, cold-start-gated, n=3 per arm unless noted)

Prefill = standalone, server-side tok/s. Decode = C1 per-request (1/ITL), MTP-3, temp 0.

• The dense quant costs −4.1 % prefill / −4.7 % decode vs the BF16-dense parent on the same image — the price of +43.7 % KV, −15 GiB disk, and the KLD floor above.
• r13 vs r11 on this checkpoint: prefill parity, decode +9.0 % (exact-m (128,128) decode-window kernels), KLD unchanged, kernel numerics bit-identical.

Full sweep on r13 (single round, results/battery/):

† n=1 anomaly, off-trend TTFT; consistent with a one-time JIT compile on a context-length kernel bucket (see traps above), re-measure pending.

Measurement discipline (why the error bars exist)

Prefill on this rig is thermally ordered, not noisy — arms alternate cold-started (<40 °C gate) in Latin-square position rotation. n=1 is never evidence; three separate campaign conclusions were reversed at n=3, and a fourth (a phantom "r13 regression") was a first-boot compile artifact caught by position rotation. Full campaign narrative and negative results (fp8/MXFP8 dense, k58 GEMV, fp16 MMA accumulation, aux-stream reconstruct prefetch — falsified at 0.3 % of end-to-end) are in the source project's REPORT.

Repo layout

launcher/                           the ACTUAL serving entrypoint chain from the
                                    image (serve-gilded-gnosis.sh -> v19 -> v16):
                                    every env var above is consumed here
compose/dense6-r13pf/compose.yml   serving compose (r13, recommended)
compose/dense6-r11pf/compose.yml   serving compose (r11, also validated)
patches/                            the two required patches + Dockerfiles
serve.sh                            bring-up + /proc env verification
results/battery/                    full bench + estonia/lavd/hotel/needle (JSON+logs)
results/tri-campaign/               the 10 position-matched speed arms
results/kld/                        KLD runs: r13 nvfp4-dyn, fp8-KV frontier, r11 canonical
results/sharegate/                  in-situ reconstruct share measurement (0.3 %)

Credits and lineage

• Z.ai for GLM-5.2.
• Brandon Music for the EXL3/TR3 encoder work, owner corpus, DENSE6 derivative construction, validation, and publication.
• turboderp for ExLlamaV3 v0.0.43 and EXL3.
• malaiwah for the MTP-78 overlay and calibration capture.
• Josh Cartu for the MTP-78 recipe and rank-sliced runtime work.
• Luke Alonso for B12X and the associated Blackwell kernel/runtime work.
• Martin Vit and yatesdr for the Infernal Invocation / RC2 image engineering credited by the upstream MTP-78 overlay.
• Special thanks to local-inference-lab for its vLLM fork, image, review, testing, and release engineering.

Canonical project attribution is also recorded in the TR3 quantization provenance.

License

Inherits the license of the base GLM-5.2 model — see the base model's terms before use. The patches and scripts in this repo are provided as-is.