GLM-5.2-NVFP4 is an NVFP4-quantized version of zai-org/GLM-5.2, a 744B-parameter Mixture-of-Experts language model with 40B active parameters, 256 experts per MoE layer (8 activated per token), and DeepSeek Sparse Attention (DSA).
Quantized directly from the full BF16 checkpoint (zai-org/GLM-5.2, not the FP8 release, to NVFP4 (4-bit with blockwise FP8 scales per 16 elements) using NVIDIA Model Optimizer.
Only the non-shared MoE expert MLP projections are quantized to NVFP4. Attention weights are left in BF16, in addition to the dense MLPs (layers 0-3) and the shared experts. Since the MoE expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings.
Calibration uses natural top-k routing rather than forcing all experts to activate, so each expert's quantization scales reflect the token distributions it actually sees during inference. To compensate, calibration was run on a much larger number of samples than typical to ensure broad expert coverage through natural routing alone.
Three calibration passes were run:
Hardware: 8x RTX PRO 6000 Blackwell 96GB (b12x MoE runner recommended)
https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm5.2_v17.md
model-mtp.safetensors 6.0 GBmodel-00013-of-00087.safetensors 5.4 GBmodel-00002-of-00087.safetensors 5.4 GBmodel-00003-of-00087.safetensors 5.4 GBmodel-00074-of-00087.safetensors 5.4 GBmodel-00050-of-00087.safetensors 5.4 GBmodel-00026-of-00087.safetensors 5.4 GBmodel-00064-of-00087.safetensors 5.4 GBmodel-00040-of-00087.safetensors 5.4 GBmodel-00016-of-00087.safetensors 5.4 GBmodel-00007-of-00087.safetensors 5.4 GBmodel-00011-of-00087.safetensors 5.4 GBmodel-00020-of-00087.safetensors 5.4 GBmodel-00030-of-00087.safetensors 5.4 GBmodel-00034-of-00087.safetensors 5.4 GBmodel-00044-of-00087.safetensors 5.4 GBmodel-00054-of-00087.safetensors 5.4 GBmodel-00058-of-00087.safetensors 5.4 GBmodel-00068-of-00087.safetensors 5.4 GBmodel-00078-of-00087.safetensors 5.4 GBmodel-00072-of-00087.safetensors 5.4 GBmodel-00036-of-00087.safetensors 5.4 GBmodel-00024-of-00087.safetensors 5.4 GBmodel-00048-of-00087.safetensors 5.4 GBmodel-00014-of-00087.safetensors 5.4 GBmodel-00061-of-00087.safetensors 5.4 GBmodel-00037-of-00087.safetensors 5.4 GBmodel-00025-of-00087.safetensors 5.4 GBmodel-00062-of-00087.safetensors 5.4 GBmodel-00038-of-00087.safetensors 5.4 GBmodel-00004-of-00087.safetensors 5.4 GBmodel-00063-of-00087.safetensors 5.4 GBmodel-00075-of-00087.safetensors 5.4 GBmodel-00051-of-00087.safetensors 5.4 GBmodel-00039-of-00087.safetensors 5.4 GBmodel-00027-of-00087.safetensors 5.4 GBmodel-00015-of-00087.safetensors 5.4 GBmodel-00005-of-00087.safetensors 5.4 GBmodel-00049-of-00087.safetensors 5.4 GBmodel-00076-of-00087.safetensors 5.4 GBmodel-00052-of-00087.safetensors 5.4 GBmodel-00028-of-00087.safetensors 5.4 GBmodel-00006-of-00087.safetensors 5.4 GBmodel-00008-of-00087.safetensors 5.4 GBmodel-00019-of-00087.safetensors 5.4 GBmodel-00031-of-00087.safetensors 5.4 GBmodel-00043-of-00087.safetensors 5.4 GBmodel-00055-of-00087.safetensors 5.4 GBmodel-00067-of-00087.safetensors 5.4 GBmodel-00079-of-00087.safetensors 5.4 GBmodel-00009-of-00087.safetensors 5.4 GBmodel-00021-of-00087.safetensors 5.4 GBmodel-00077-of-00087.safetensors 5.4 GBmodel-00018-of-00087.safetensors 5.4 GBmodel-00022-of-00087.safetensors 5.4 GBmodel-00023-of-00087.safetensors 5.4 GBmodel-00032-of-00087.safetensors 5.4 GBmodel-00033-of-00087.safetensors 5.4 GBmodel-00041-of-00087.safetensors 5.4 GBmodel-00042-of-00087.safetensors 5.4 GBmodel-00046-of-00087.safetensors 5.4 GBmodel-00053-of-00087.safetensors 5.4 GBmodel-00056-of-00087.safetensors 5.4 GBmodel-00065-of-00087.safetensors 5.4 GBmodel-00069-of-00087.safetensors 5.4 GBmodel-00070-of-00087.safetensors 5.4 GBmodel-00010-of-00087.safetensors 5.4 GBmodel-00017-of-00087.safetensors 5.4 GBmodel-00029-of-00087.safetensors 5.4 GBmodel-00045-of-00087.safetensors 5.4 GBmodel-00057-of-00087.safetensors 5.4 GBmodel-00066-of-00087.safetensors 5.4 GBmodel-00080-of-00087.safetensors 5.4 GBmodel-00035-of-00087.safetensors 5.4 GBmodel-00047-of-00087.safetensors 5.4 GBmodel-00059-of-00087.safetensors 5.4 GBmodel-00071-of-00087.safetensors 5.4 GBmodel-00086-of-00087.safetensors 5.4 GBmodel-00073-of-00087.safetensors 5.4 GBmodel-00060-of-00087.safetensors 5.4 GBmodel-00082-of-00087.safetensors 5.4 GBmodel-00083-of-00087.safetensors 5.4 GBmodel-00084-of-00087.safetensors 5.4 GBmodel-00001-of-00087.safetensors 5.3 GBmodel-00012-of-00087.safetensors 5.2 GBmodel-00081-of-00087.safetensors 4.2 GBmodel-00087-of-00087.safetensors 3.6 GBmodel-00085-of-00087.safetensors 2.7 GBmodel-inputscales.safetensors 7 MBmodel-mtp-inputscales.safetensors 86.2 kB有文件？提交种子。
提交种子
LocalMaxxingvllm serve /model --served-model-name GLM-5.2 --tensor-parallel-size 4 --quantization modelopt_fp4 --attention-backend B12X_MLA_SPARSE --moe-backend b12x --kv-cache-dtype fp8 --max-model-len 1000000 --gpu-memory-utilization 0.95 --max-num-seqs 2 --enable-chunked-prefill --enable-prefix-caching --async-scheduling --speculative-config {"model":"/model","method":"mtp","num_speculative_tokens":5} --host 0.0.0.0 --port 5001在 LocalMaxxing 上查看测试 来自 LocalMaxxing 的 3 次社区测试（1 种配置）的中位数。
自己运行：配置指南