GLM-5.3-Flash for mlx-serve, iQ-MLX 5.1 bpw (GPTQ experts, 4/5/6-bit by layer)

mlx-serve pack of zai-org/GLM-5.3-Flash (model_type: glm5_next, 320B total / 18B active). It takes 202.5 GB on disk and runs on a stock 256 GB Mac. The pack includes the MTP head and the vision tower.

The routed experts are quantized with GPTQ on Hessians collected from the BF16 model. The bits go where errors grow the most, in the early layers. At 9% more bytes than the 4-bit reference pack, the KLD to BF16 is half, and decode speed is the same. It matches the 224 GB 5-bit oQ5e pack's quality in 21.5 GB less, decodes about 10% faster in mlx-serve, and fits without raising macOS's GPU memory limit.

Download MLXServe.com

mlx-serve --model ddalcu/GLM-5.3-Flash-MLX-Serve-iQ-MLX-5.1bpw --serve

Calibration and what it buys

The BF16 checkpoint was streamed layer by layer over 1.05 M calibration tokens: agent transcripts, code, prose, math and Wikipedia. Each expert gets its own Hessians, and every token's input is weighted by its router weight. GPTQ uses damping 0.1 and group size 64.

Quality is scored against the BF16 model's own logits on 31 held-out windows of 2048 tokens. Top-1 means the pack's greedy token equals BF16's. KLD is KL(BF16 || pack) over BF16's top-1024 tokens.

Scores cover assistant-turn and raw-text positions only. Inside a chat user turn, GLM-5.3 predicts its own answer rather than the user's text, and those positions flip on any perturbation, BF16 numerics included.

The table above uses all 25,471 assistant and raw-text positions. Each pack's own weights were dequantized and run layer by layer against the BF16 model. By region, this pack is ahead of oQ5e on assistant turns (KLD 0.0403 vs 0.0416), and oQ5e is ahead on raw Wikipedia text (0.0466 vs 0.0496).

The same comparison served by mlx-serve, on 392 sampled positions:

On those positions the served KLD for this pack matches its harness KLD: median difference 0.0001, and no position is more than 0.6 nats apart.

For scale: a single 8-bit tensor in layer 0, with everything else BF16, already lands at top-1 94.3% / KLD 0.025. Early perturbations change routing downstream. That is why non-expert upgrades buy nothing here, and the bytes go to the routed experts.

Speed on an M5 Ultra (256 GB), two 1024-token greedy answers per pack (code, prose), decode tok/s:

At macOS's default GPU memory limit (222.7 GiB on a 256 GB Mac), oQ5e's 208.5 GiB of weights does not leave room to load: mlx-serve is killed during the load and oMLX aborts the first request. This pack loads with room to spare in both.

Bits per weight, scales included: 5.05 over the 320.6 B quantized weights.

Widths

Every quantized tensor uses group size 64. The per-layer plan was chosen from measured sensitivity: expert error at layers 3-10 costs about 10x the KLD of the same error at layers 31-44, and down_proj costs about 1.6x more KLD per byte than gate/up. Gate and up share a width in every layer, as mlx-serve's fused MoE decode kernel requires.

Compatibility

Tensor names and layout follow the oQ4-MTP pack, and the config keeps the upstream per-layer lists (45 entries, none for the MTP layer). Tested loading and answering on an M5 Ultra:

mlx-lm has no glm5_next model yet. The architecture lives in mlx-vlm (0.7.1 and later; oMLX 0.7.1.dev1 ships 0.7.7), which oMLX uses to load it.

Serving notes

• Memory. About 203 GB resident plus KV cache, so it needs a 256 GB Mac. mlx-serve sizes the context to what fits.
• MTP. The checkpoint's own mtp.0 layer drafts by default (--no-mtp turns it off).
• Thinking is on by default ("enable_thinking": false turns it off). Tools use GLM's <arg_key> call format.
• Images go through the GLM vision tower (<|image|> rows).

Conversion

The conversion scripts are a one-off layer-streamed pipeline and are not published. It runs a reference forward per layer, collects router-weighted per-expert Hessians, runs batched GPTQ, and writes the pack in oQ4-MTP's layout. The MTP head and the vision tower are copied from oQ4-MTP. Peak memory is about 60 GB; the run took 2.3 hours on an M5 Ultra.