GLM-5.3 Int4/Int8 mixed-precision AWQ, group 64

A mixed-precision quantization of GLM-5.3 (754B-parameter MoE, model type glm_moe_dsa) made from the original BF16 release (zai-org/GLM-5.3-BF16, not the derived FP8 checkpoint). Built for the DGPP inference engine on four NVIDIA DGX Sparks (TP=4), where the whole model has to fit in ~100 GiB per rank with room left for context.

Method

• AWQ (activation-aware weight quantization) via llm-compressor 0.13.0, applied one decoder layer at a time by a layer-wise driver that is bit-exact against llm-compressor's single-pass sequential pipeline (verified on a reduced model, both group sizes). Activations propagate through the smoothed, unquantized layer, as in the reference implementation.
• Calibration: 512 sequences × 2048 tokens (1.05M tokens) drawn with a fixed seed from a mixed corpus of prose, code, math and conversational / tool-use transcripts. Each routed expert is calibrated on the tokens actually routed to it. A second, independent 512-row draw reproduced the layer output error to within 0.1 %, so the calibration size is not a limiting factor.
• Fold correction (important). llm-compressor's AWQ mapping for glm_moe_dsa folds the smoothing scale of post_attention_layernorm into the norm but does not rescale the MoE router (mlp.gate), and likewise leaves the DSA indexer's wk / weights_proj / wq_b uncompensated in the 18 layers that own an indexer. As saved by llm-compressor those modules receive x / s, and the per-layer error is 5–10× worse than plain round-to-nearest. Every MoE shard here has the affected weight columns rescaled by the recovered scale (original_norm / smoothed_norm), which restores the layer to slightly better than round-to-nearest. Corrected shards carry the safetensors metadata key fold_corrected. The defect and fix are described in the issue draft shipped with this card's tooling.

Measured error (per layer)

Relative RMS error against the bf16 layer on 32 × 2048 real, held-out tokens. "Contribution" is the error of what the layer adds (output minus input); "residual" is the error of the full residual stream leaving the layer, which is the quantity that propagates.

The contribution metric grows with depth because each layer's addition shrinks relative to the residual stream; the residual error rises from 0.5 % (layer 3) to 2.3 % (layer 63). On this model the int4/int8 mix lands at the error of an all-fp8 recipe at half the bytes. AWQ's advantage over round-to-nearest is 0.9 / 4.8 / 1.2 / 0.4 / 0.02 % of the error at layers 3 / 6 / 21 / 42 / 63: real in the shallow indexer layer, negligible at depth. Per layer, this checkpoint and the round-to-nearest companion are equivalent.

No end-to-end evaluation yet (perplexity, benchmarks): the numbers above are per-layer. Treat this as a carefully validated but not yet benchmarked checkpoint.

A companion checkpoint with the same recipe but plain round-to-nearest instead of AWQ (no smoothing, no fold correction needed) is at HawkBearPig/GLM-5.3-Int4-Int8Mix-RTN-g64.

Loading

Standard compressed-tensors layout: model.safetensors.index.json, per-layer shards layer-NNN.safetensors, passthrough.safetensors (embeddings, norms, lm_head, MTP block), config.json with quantization_config, tokenizer files. Any loader that understands glm_moe_dsa and compressed-tensors pack-quantized W4A16/W8A16 should read it; only DGPP has been exercised. vLLM / transformers loading is untested.

Tooling: llm-compressor 0.13.0, compressed-tensors 0.18.0, transformers 5.14.1, torch 2.13.0 (CUDA 13), one B200.

License: MIT, inherited from GLM-5.3.