GLM-5.2-REAP-504B-GGUF — BF16 + dynamic 2/3/4-bit

GGUF builds of the 34%-expert-pruned, Router-KD-recovered GLM-5.2-504B — for llama.cpp (CPU / Metal / CUDA). Includes the first working GGUF of a GLM-5.2 DSA model, made possible by a custom patch for its shared sparse-attention indexer.

The full-precision NVFP4 model lives at 0xSero/GLM-5.2-504B — start there for the model story, eval, and architecture. This card is about the GGUF conversion specifically.

---

🙏 Sponsor

Built on 8× NVIDIA B200 sponsored by Lambda. Thank you. 🙏

---

Files

Files > 45 GB are split (...-00001-of-000NN.gguf); point llama.cpp at the first shard and it loads the rest automatically.

What "dynamic" means here

These are not flat K-quants. Each uses per-tensor precision: the parts that hurt most under quantization are kept high while the bulk (routed experts) is pushed low:

• token-embedding → Q6_K, output head → Q8_0/Q6_K
• attention (MLA q/kv) → Q6_K / Q5_K / Q4_K (by tier)
• shared expert → Q8_0 / Q6_K
• routed-expert down-projection kept above gate/up (it's the sensitive one)

Not yet imatrix-calibrated. A true importance-matrix pass needs the model running under llama.cpp, and our llama.cpp build is CPU-only while the GPUs were committed elsewhere. The per-tensor recipe still puts these well above naive quants; imatrix-calibrated builds may follow.

The hard part — making GLM-5.2 loadable in llama.cpp

GLM-5.2 uses DSA (DeepSeek-style sparse attention) with a shared indexer: only ~1 layer in 4 is a "full" indexer layer; the other 57 of 78 layers share it and carry no indexer weights of their own (index_topk_freq: 4). Stock llama.cpp demands an indexer tensor on every layer, so the converted GGUF failed to load:

missing tensor 'blk.3.indexer.k_norm.weight'

The fix (applied to these files): we filled each of the 57 shared layers' indexer tensors (285 tensors total) by duplicating them from the nearest preceding full layer, making the GGUF loadable.

The MTP / next-token-prediction layer (blk.78) is preserved — kept at Q6_K in every quant (the stock quantizer rejects the nextn layer's experts, so we pin them with explicit per-tensor types). This keeps self-speculative decoding available in llama.cpp for this architecture.

Verified: the patched model loads and generates coherent text end-to-end in llama-cli.

Caveat (honest): the sharing is approximated — the duplicated weights are recomputed per layer rather than the index being reused exactly as the original does. Output is coherent in testing, but this is not a bit-exact reproduction of the reference attention. Prefer Q4_K_XL or BF16 where fidelity matters most; the low-bit quants were not individually loop-tested.

Run it

# loads all shards from the first one
llama-cli    -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf -p "Hello"
llama-server -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf --host 0.0.0.0 --port 8080

Tips (measured at n=2000): a sampler guardrail is the single best free anti-loop knob — --min-p 0.05 --repeat-penalty 1.05 → loop 7.2%→4.9%; --repeat-penalty 1.10 → 2.3%, i.e. it fully recovers the pruning-induced looping (raw 7.2%; unpruned teacher 3.6%). Use 1.05 by default, 1.10 if you see loops (higher penalty can slightly over-suppress legitimate repetition in code). A "Be concise…" system prompt separately halves output length (it shortens, doesn't de-loop).

How the underlying model was made (short version)

REAP expert-pruning (gate × ‖expert_output‖) → keep 168/256 experts/layer (~504B, 34% pruned) → gate-only Router-KD to the unpruned teacher (train 0.016% of params). On n=2000 held-out real prompts it loops at 7.2% vs the unpruned teacher's 3.6% — pruning roughly doubles the loop rate (gate-KD recovers routing but not full termination; a knowledge-recovery LoRA is in progress to close the gap). Full details + honest accounting on the NVFP4 model card.

---

GGUF conversion + custom DSA-indexer patch. Compute sponsored by Lambda — thank you. 🙏