GLM-5.2-NVFP4-REAP-Recall (N=172)

A recall-recovered re-REAP of GLM-5.2-NVFP4 — 172 experts/layer (from 256), self-consistent, BF16 weights preserved, NVFP4 experts sliced byte-for-byte.

• Recipe + serving stack: github.com/brandonmmusic-max/GLM-5.2-Reap
• Docker image: verdictai/glm52-nvfp4-dcpmtp:v3.3
• Saliency methodology + corpus ledger: REAP_RECALL_VERDICT.md
• Verified launch script: serve_glm52_reap_recall.sh

Attribution: GLM-5.2 by Z.ai · NVFP4 parent by Luke Alonso · REAP by Cerebras Research (arXiv:2510.13999, ICLR 2026).

---

Why this exists

Standard REAP compresses GLM-5.2 well for code, agentic, and reasoning workloads — but closed-book factual recall collapses: the published narrow-calibrated REAPs answer Lexington for the capital of Kentucky and loop on Marbury v. Madison. The cause is not the model and not the serving stack — it's a calibration corpus that excludes knowledge. This checkpoint re-runs the REAP saliency on a knowledge- and legal-inclusive, axis-balanced calibration.

All reasoning prompts still pass (8/8 traps; pipes/syllogism/discount/anticipatory-repudiation all finish=stop).

---

What I did specifically

1. Diagnosed it as calibration, not the model — A/B'd kernels, MTP, DCP, image, sampling. Same prompts, same parser, same image: behavior tracked the calibration corpus, not anything else.
2. Built a 4-axis balanced calibration — 12,228 samples, 3,057 per axis, max 16,384 tokens, no truncation, no packing:
   • Axis 1 — General knowledge: C4, Wikipedia, MMLU-aux, TriviaQA, Natural Questions
   • Axis 2 — Legal: 1,528 CAP markdown cases + a live Neo4j legal KG (300 headnotes, 390 statutes, 373 case summaries, 113 fact-atoms, 353 worked-examples). fallback_used: false.
   • Axis 3 — Code/agentic: evol-codealpaca, Magicoder, xLAM, SWE-smith
   • Axis 4 — Reasoning/termination: terminating <think>…</think> traces, </think>-region weighted ×6
3. Built a real block-wise NVFP4 saliency runner — the published REAP loader can't consume glm_moe_dsa + modelopt NVFP4, and a whole-model vLLM load OOMs the intact 435 GB before any forward. The runner chunks decoder layers into VRAM, dequants NVFP4 → BF16 in place, runs the GlmMoeDsaNaiveMoe modules (explicit per-expert outputs — the ideal saliency hook), captures S_j = mean_{active tokens}(router_gate_j · ||expert_output_j||₂), frees the chunk. Real GPU saliency over 7,368,253 active tokens across 75 MoE layers — no static proxy.
4. Self-consistent prune at prune time — kept experts renumbered contiguous 0…171, router shrunk to [172, 6144], bias to [172], n_routed_experts = num_experts = 172. Loads clean on stock vLLM with no repair_reap.py.

---

Quick start

# Pull serving image
docker pull verdictai/glm52-nvfp4-dcpmtp:v3.3

# Download model (294 GB)
huggingface-cli download brandonmusic/GLM-5.2-NVFP4-REAP-Recall-N172 \
  --local-dir $HOME/models/GLM-5.2-NVFP4-REAP-Recall-N172

# Serve on 4x RTX PRO 6000 96GB (sm120)
docker run -d --name glm52-reap-recall \
  --gpus all --runtime nvidia --ipc host --shm-size 32g --network host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$HOME/models":/models-archive:ro -v "$HOME/.cache/glm52-b12x":/cache \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e CUTE_DSL_ARCH=sm_120a \
  -e HF_HUB_OFFLINE=1 -e NCCL_IB_DISABLE=1 -e NCCL_P2P_LEVEL=SYS -e NCCL_PROTO=LL,LL128,Simple \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e B12X_MHC_MAX_TOKENS=16384 -e VLLM_USE_B12X_WO_PROJECTION=1 -e VLLM_USE_B12X_MHC=1 \
  -e VLLM_USE_B12X_FP8_GEMM=1 -e VLLM_USE_B12X_MOE=1 -e VLLM_USE_B12X_SPARSE_INDEXER=1 \
  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_FUSED_MOE_GROUPED_TOPK=1 \
  -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x -e VLLM_ENABLE_PCIE_ALLREDUCE=1 \
  -e B12X_MLA_SM120_UNIFIED=1 -e USES_B12X=True -e B12X_DENSE_SPLITK_TURBO=1 -e B12X_W4A16_TC_DECODE=1 \
  -e B12X_MOE_FORCE_A16=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DCP_SHARD_DRAFT=1 \
  verdictai/glm52-nvfp4-dcpmtp:v3.3 \
  python -m vllm.entrypoints.cli.main serve /models-archive/GLM-5.2-NVFP4-REAP-Recall-N172 \
    --served-model-name glm-5.2-nvfp4 --host 0.0.0.0 --port 9405 \
    --kv-cache-dtype fp8 --block-size 256 --load-format safetensors \
    --tensor-parallel-size 4 --decode-context-parallel-size 4 --moe-backend b12x --linear-backend auto \
    --gpu-memory-utilization 0.92 --max-model-len 200000 --max-num-seqs 16 \
    --enable-chunked-prefill --enable-prefix-caching --max-num-batched-tokens 8192 \
    --max-cudagraph-capture-size 64 --attention-backend B12X_MLA_SPARSE \
    --compilation-config '{"custom_ops":["all"],"cudagraph_mode":"PIECEWISE"}' \
    --enable-flashinfer-autotune \
    --hf-overrides '{"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"probabilistic","moe_backend":"b12x","use_local_argmax_reduction":true}'

# Sanity check
curl -s http://127.0.0.1:9405/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"glm-5.2-nvfp4","messages":[{"role":"user","content":"what is the capital of kentucky?"}]}'

Optional — thinking_token_budget (hard-caps the reasoning loop): mount the four V2 patch files from the repo into the container and add --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}', then pass "thinking_token_budget": N in the request body. Without those patches, leave --reasoning-config off — on a plain image it clobbers glm45's <think> priming and the chat path stops thinking.

---

Verified serving config (4× RTX PRO 6000 96GB, TP4)

The v3.3 serving config (image verdictai/glm52-nvfp4-dcpmtp:v3.3):

---

Benchmarks (measured on 4× RTX PRO 6000 96GB)

Note: the numbers below were measured on the prior nvfp4_ds_mla (4-bit MLA KV) config. A v3.3 (fp8) re-bench is pending.

Benchmarked with local-inference-lab/llm-inference-bench, concurrency × context grid. 15/15 cells completed.

Decode throughput (tok/s)

Real-prompt single-user (Marbury essay × 5, thinking OFF)

Prefill (full ingest)

KV pool: 542,857 tokens · * VRAM peak: 97.93%

---

Files

• 87 safetensors shards (294 GB), config.json, model.safetensors.index.json, tokenizer files, generation_config.json, chat_template.jinja
• reap_recall_keep_map_with_scores.json — per-expert real saliency scores and the kept-expert map (the replication artifact)
• REAP_RECALL_VERDICT.md — full corpus / saliency / validation ledger
• serve_glm52_reap_recall.sh — verified launch script

---

Sampling

{
  "temperature": 1.0,
  "top_p": 0.95,
  "repetition_penalty": 1.05,
  "stop_token_ids": [154820, 154827, 154829],
  "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "high"}
}

For non-thinking generation, set enable_thinking: false. For Marbury-style essays, max_tokens >= 1500 recommended (thinking-high needs room).

---

Credits and lineage

Attribution to the sources:

• Z.ai — GLM-5.2 base.
• Luke Alonso — GLM-5.2-NVFP4 parent checkpoint and B12X kernels used by the documented serving profile.
• NVIDIA TensorRT Model Optimizer — NVFP4 format/tooling consumed by the parent and the block-wise saliency adaptation.
• Cerebras Research — REAP method and code, paper, and OpenReview record.
• Brandon Music — recall-inclusive calibration corpus, GLM/ModelOpt adaptation, saliency run, self-consistent pruning, validation, and publication.
• The local-inference-lab community — Blackwell serving, testing, and runtime engineering.

@inproceedings{lasby2026reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=ukGxWd2aDG}
}

License

MIT. See the upstream model and source-project licenses for their respective terms.