GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP

Model description

INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 MTP draft head kept for speculative decoding. Only the 36,288 routed-expert GEMMs are INT4 (GPTQ, symmetric, group-size 128); attention, router, shared experts, embeddings, the vision tower and the MTP head stay in BF16. Not a new model — all capability comes from the base model.

Full benchmark grids, comparison protocols and research notes: BENCHMARKS.md.

Uses & recommended recipes

Quick start

# 1. Download (~178 GiB)
huggingface-cli download canada-quant/GLM-5.3-Flash-W4A16-MTP --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve on 4× H100 / H200 (other hardware: see Serving recipes)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Two things that bite. If your config.json predates 2026-09-08, re-download it — older copies fail in vLLM with KeyError: 'layers.0.mlp.gate_up_proj.weight' (weights are unchanged). And always pass --max-num-seqs ≤ 512 — the vLLM default of 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.

Serving recipes

All configurations use expert parallelism. fp8 KV is not available on Hopper for this NoPE model.

H100 / H200, TP=4 (pinned image) — the Quick start command is the benchmarked recipe. num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance; N=5 collapses acceptance to ~30%). Keep prefix caching off — it measured −2…−5% on H200. On 8× H200, two independent TP=4 replicas behind a load balancer beat one TP=8 endpoint by +28–38% aggregate at c128–c512; TP=8 wins single-stream and holds one 7.8M-token pool. On 2× H200 (89.5 GiB weights per GPU) add --tensor-parallel-size 2 --max-num-seqs 128 --max-cudagraph-capture-size 128 for 262K, or --max-model-len 1048576 --max-num-seqs 16 --gpu-memory-utilization 0.95 --max-cudagraph-capture-size 64 --max-num-batched-tokens 4096 for 1M.

8× H100 + DFlash2-G (v2 image, 2026-09-28 closeout) — latency recipe = TP8 + DFlash2-G K=7: c1 181.2 tok/s, +46% vs spec-off (124.1). Aggregate recipe = TP8 spec-off or MTP-N2: c256 2,418 / 2,330 tok/s; TP8 MTP-N2 is the balanced middle with the best grid c1 (215.0). Do not run DFlash2 at TP4 past c128 — the drafter's 8 standalone FullAttention KV tensors cut the KV pool ~6× and aggregate collapses (structural, confirmed uncontended: 694 tok/s @c128 vs TP8's 2,220); MTP-N2's shared-embedding draft softens the cliff (solo-TP4 c1 193.2, the best single-stream shape measured on this rig, c256 1,292). With any drafter on this image, cap --max-model-len ≤ 131071 — 128K-token prefills die on the speculative path (spec-off survives them). Full grids: BENCHMARKS.md. On the same rig and harness, NVIDIA's own NVFP4 checkpoint — an emulated FP4 path on Hopper (no native SM90 FP4 kernels; the engine warns at boot) — delivers 122.56 tok/s c1 / 2,299.84 @c256 at its best arm against our 181.2 / 2,418.1: on Hopper, use our quant + drafter + recipe (full head-to-head).

8× H100, TP=8, DFlash2-G (bench-validated 2026-09-28) — single node, 8192/1024 prompts: the latency recipe is TP=8 + DFlash2-G K=7: 181.2 tok/s c1, +46% vs spec-off 124.1; the aggregate recipe is TP=8 spec-off: 2,418 tok/s @c256 (MTP N=2 at 2,330 is the balanced middle and the best c1 at 215.0). K=4 beats K=7 on aggregate at every concurrency. Avoid TP=4 with the DFlash2 drafter past c32: its 8 standalone full-attention KV tensors cut the KV pool ~6× vs TP8 (311,999 vs 1,811,949 tokens) and force a preemption/recompute TTFT cliff that is structural, not contention — solo-TP4 measured 160.5 K=7 / 165.7 K=4 / 123.9 spec-off / 193.18 MTP-N2 tok/s c1, MTP-N2 being the best single-stream shape measured on this rig at any TP (1,292.4 @c256, the only speculative arm still rising there). With any drafter, cap --max-model-len ≤ 131071 — the speculative path faults on 131K-token prefills (spec-off survives); image ghcr.io/canada-quant/vllm-glm53-flash-h100:v2-w4a16-dflash2, cold boot ≈ 12 min.

RTX PRO 6000, TP=4 — same command with the SM120 image and --max-num-seqs 64 --max-num-batched-tokens 8192 --kv-cache-dtype fp8 --enable-prefix-caching. fp8 KV is required at 262K on 96 GB cards. Keep MTP on at every concurrency here: it adds +70% at c1 and +48% at c32.

2× DGX Spark, TP=2, 1M context — prebuilt image and one-command launcher in canada-quant/vllm-glm53-flash-sm121; the launcher also ships in the drafter repo. Drafter: canada-quant/GLM-5.3-Flash-DFlash2-G (the authors' self-trained DFlash2 drafter, Apache-2.0; 3.676 mean acceptance at K=7 on the 500-prompt holdout vs 3.632 for the incoai reference measured on the same hardware; drop-in successor of -F and -E, E having carried the 2026-09-22 banked Spark row and G being the Spark drafter of record since 2026-09-25 (the 2026-09-27 H2H champion leg and the live production serve both ran G, sha256-verified) — DRAFTER_HOST_PATH=/models/GLM-5.3-Flash-DFlash2-G). Start the worker rank first, wait 25 s, then the head rank.

# on both nodes, rank1 (worker) first, then rank0 (head) 25 s later
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 GMU=0.90 EAGER=0 GRAPHS=1 bash launch_dflash2_tp2.sh <rank>

Hard constraints (Spark): num_speculative_tokens must be 7 (any other count wedges boot); confirm the boot log shows the mask-embedding load (mask_token_id 154856); keep single prompts ≤ ~310K tokens; stop with docker stop -t 30, never rm -f. Cold boot is 6–10 minutes.

Quality

On RTX PRO 6000, ≈63% of the raw AIME 2025 deficit is a budget wall (empty-answer rate 11.7–14.2% vs 3.3%) and ≈37% is SM120 kernel numerics; a zero-cost commit hook closes it but is not part of the published recipes. Vision (formal, 2026-09-28): MMMU validation 0.74667 (672/900) and OCRBench 888/1000 on the 8×H100 closeout stack (lmms-eval 0.7.3 task specs, greedy temp 0, seed 42). The vision tower is BF16 passthrough, not covered by the text-only calibration — these are the artifact's own vision baseline; the only same-rig vision baseline measured to date is the NVIDIA NVFP4 checkpoint (MMMU 0.68 / OCRBench 882, 2026-09-28). Details in BENCHMARKS.md.

Benchmarks summary

Output tok/s, thinking ON, same hardware, flags and prompts within each row.

vs NVIDIA's NVFP4 checkpoint — 8× H100, same rig (2026-09-28)

nvidia/GLM-5.3-Flash-NVFP4 is a Blackwell-only release and Hopper has no native FP4 math: on SM90 every weight dequantizes through FP4-emulation kernels (the engine warns at boot) — an emulated path, not a native comparison. Measured on the same p5 rig and closeout harness (8192/1024, thinking ON). The card-verbatim fp8-KV recipe cannot boot on Hopper (scale item not float32 in the KV-quant kernel); the card-recipe arm runs with KV bf16, the one documented deviation. Output tok/s:

Our recipes lead NVFP4's best arm (N2) by 47.8% at c1 (181.2 vs 122.56; solo MTP-N2 193.18 = +57.6%) and 5.1% at c256 (2,418.1 vs 2,299.84); c128 is a dead heat (2,066.8 vs 2,081.13, NVFP4 +0.7%). The card-recipe arm (N1b) collapses under load — TP4 KV exhaustion drives TTFT p50 to 263 s @c256 — and the checkpoint ships no MTP layer, so it cannot express GLM's native speculative decode at all. On Hopper, use our quant + drafter + recipe. Boot-failure autopsy and sha256 custody: BENCHMARKS.md.

Single-stream decode is insensitive to KV length up to ≥486K (RTX PRO 6000); at batch, long-KV decode plateaus at ~2–4 tok/s per stream and long prefills serialize at a ~6–8.5K tok/s aggregate ceiling. All grids, protocols and the H200 extended table: BENCHMARKS.md.

Hardware & context limits

Each row is the largest context serving-validated on that configuration.

Known issues

• config.json (2026-09-08): vLLM matches quantization_config.ignore against its own fused module names, so the ignore list now carries both the HF and vLLM spellings plus re:.*\.layers\.45\..* for the MTP head. Older 765-entry copies fail at load. Weights unchanged.
• DFlash2 admission wedge (SM90 research stack only, MTP recipes unaffected): with the DFlash2 drafter at block size 2304, prompts above ~15.5K tokens are never admitted. A fix was validated to 256K prompts; block size 1536 avoids it. Filed as vllm-project/vllm#55800.
• Marlin no-split-K path on SM121: deterministic illegal memory access at M=256 when forcing split_k=1; the stock heuristic used in serving is clean. Filed as vllm-project/vllm#56064.

Quantization details

Build gates, all passing: exactly 36,288 packed tensors and nothing quantized outside routed experts; vision key set 348/348 identical to source; MTP layer present; zero dtype drift vs source; no collapsed expert scales. Loads with transformers ≥ 5.16; text generation and image captioning smoke tests pass.

Citation & license

@misc{canada_quant_glm53_flash_w4a16_mtp,
  title  = {GLM-5.3-Flash W4A16 (INT4) + BF16 MTP},
  author = {canada-quant},
  year   = {2026},
  url    = {https://huggingface.co/canada-quant/GLM-5.3-Flash-W4A16-MTP}
}

MIT, inherited from the base model. Follow the base model's usage terms.

---

Built, benchmarked and documented with the Digby.ai coding harness, developed by CQL.ca.