GLM-5.3-Flash — MLX NVFP4 hybrid with embedded MTP

An Apple-silicon (MLX) packaging of Z.ai's GLM-5.3-Flash, built from NVIDIA's NVFP4 checkpoint nvidia/GLM-5.3-Flash-NVFP4. It targets the Darkbloom native glm5_next runtime. The text model, the vision tower and the multi-token prediction (MTP) layer are all in one indexed checkpoint: one download. The MTP head (layer 45, 2,641 arrays) ships in these shards, so no separate draft model or MTP file is needed.

Status: qualified for single-Mac serving on the Darkbloom network (catalog id glm-5.3-flash, 128K context, text, image and video input). Measured on one M5 Ultra (256 GB) with the Darkbloom native runtime, embedded MTP on by default: prefill 1,897 / 2,042 / 2,042 tok/s at 2.7K / 10.5K / 31.6K prompt tokens, about 1,660 tok/s at 128K; decode 62 / 59 tok/s for 256 / 1,024 tokens at one request (MTP acceptance about 0.6–0.8 per proposed token). Needle retrieval passes at 32K, 60K, 128K and 190K tokens. The record is docs/reports/2026-10-21-glm53-flash-single-mac-qualification.md in Layr-Labs/d-inference.

Formerly EigenLabs/GLM-5.3-Flash-MLX-4bit-mtp; the old name redirects here. This repository previously held a uniform affine 4-bit build (commit efb9557e38d2b6e6bbec1b514b19b529f3fef42e). That revision stays available by commit hash; the current files replace it.

What is in it

Routed experts are NVFP4 and the remaining matrices affine 4-bit; routers, norms, scalar and state parameters and the whole vision tower stay in source precision.

NVFP4 and affine g64 both cost 4.5 bits per weight, so decode bandwidth matches the earlier all-affine build while the experts keep NVIDIA's calibrated NVFP4 values. NVIDIA reports BF16-level accuracy for its NVFP4 checkpoint (for example GPQA Diamond 0.9211 vs 0.9217 BF16, MMMU-Pro 0.763 vs 0.769). Those numbers are for NVIDIA's checkpoint on NVIDIA GPUs; the extra 4-bit quantization of the non-expert matrices here is ours and has not yet been measured end to end.

NVFP4 storage

For each NVFP4 module <base>:

W = e2m1(code) × e4m3(scale) × weight_scale_2. NVIDIA's ModelOpt U8 packing and MLX's U32 packing are the same byte stream, so .weight, .scales and .weight_scale_2 are byte-for-byte NVIDIA's tensors (108,891 tensors, SHA-256 matched per tensor). NVIDIA's activation input_scale values (36,297) are GPU-only and dropped. config.json declares the default {"bits": 4, "group_size": 64, "mode": "affine"} and one {"bits": 4, "group_size": 16, "mode": "nvfp4"} override per NVFP4 module.

Note for other MLX loaders: mx.dequantize(..., mode="nvfp4") without global_scale returns e2m1 × e4m3; multiply by weight_scale_2. MLX's own global_scale argument means amax (the kernel multiplies by global_scale / 2688), which is not the stored quantity.

Affine non-expert rounding

The non-expert matrices use the same deterministic CPU recipe as the earlier EigenLabs affine build: for each group of 64 weights, MLX round-to-nearest plus 74 clipped-range candidates with 2 least-squares scale/bias refinements, lowest squared error wins (RTN on ties). The sampled outputs are byte-identical to the previous build's tensors. Relative Frobenius error against BF16 (weight statistics only, not a quality measurement):

Multi-token prediction (MTP)

• Embedded: the trained MTP layer is in this checkpoint as model.language_model.layers.45.* (num_nextn_predict_layers: 1 in config.json): enorm, hnorm, eh_proj, shared_head.norm, sparse attention with indexer, router, 288 routed experts and a shared expert. It reuses the target model's embedding and output head. There is no separate draft model to download.
• Darkbloom: the native glm5_next runtime serves GLM-5.3-Flash with MTP on by default; no flags or environment variables are needed. Proposed tokens are always verified by the target model before they are emitted.
• Other MLX runtimes: stock mlx-lm / mlx-vlm have not been evaluated with this checkpoint and are not expected to load it unmodified (per-expert tensor names instead of stacked switch_mlp, NVFP4 per-tensor scales, MTP stored as layer 45). Most generic loaders drop or ignore MTP weights.
• vLLM / SGLang (NVIDIA GPUs): use NVIDIA's original checkpoint nvidia/GLM-5.3-Flash-NVFP4, not this MLX repack, and enable MTP with --speculative-config '{"method":"mtp","num_speculative_tokens":N}' (pick a small N such as 1–3).

MTP acceptance on these weights has not been measured yet. The MTP experts were re-quantized from BF16 (RTN nvfp4, mean relative weight error 0.0946), so acceptance may differ slightly from the BF16 MTP layer.

Chat template

chat_template.jinja is Z.ai's template from zai-org/GLM-5.3-Flash-BF16 at a5b45eb4 with four edits for the runtime's Swift Jinja (lines 81, 85, 103, 234: x.0 → x[0]); under Jinja2 the two compile to identical code.

Tokenizer, tokenizer configuration, processor and generation configuration are byte-identical to zai-org/GLM-5.3-Flash-BF16@a5b45eb4 (the tokenizer and processor files are also identical to NVIDIA's). config.json is Z.ai's configuration plus quantization / quantization_config.

Download

from huggingface_hub import snapshot_download

model_path = snapshot_download(
    repo_id="EigenLabs/GLM-5.3-Flash-MLX-NVFP4-mtp",
    revision="<full commit hash>",
)

Pin a full commit hash for reproducible deployments.

Validation and limitations

• Source integrity: all 33 NVIDIA weight shards were verified against their published LFS SHA-256 at da920bb0b9f4a06727223a349e55468e38352348 before conversion; metadata files were verified against a pinned API snapshot.
• Lossless NVFP4 repack: per-tensor SHA-256 of every repacked tensor equals NVIDIA's. For 35 sampled modules (routed experts across layers and projections, all dense MLP projections, MTP experts) MLX 0.32.2 dequantize(mode="nvfp4") and a NumPy implementation of ModelOpt's formula agree bit for bit.
• Semantics check: NVIDIA's NVFP4 dense-MLP weights dequantize to within 0.090–0.094 relative error of Z.ai's BF16 weights (a swapped nibble order would give 1.41+).
• BF16 agreement: every BF16 non-expert tensor in NVIDIA's checkpoint equals Z.ai's BF16 weights (byte-identical; NVIDIA's BF16 copies of hc_*_base/scale are exact BF16 roundings of the F32 originals, which are kept here; NVIDIA's F32 conv1d weights are exact upcasts of the BF16 originals, kept as BF16). Sampled MTP expert weights are byte-identical to Z.ai's BF16 weights.
• Native load and serving (2026-10-21, one M5 Ultra 256 GB): the native loader verifies every manifest file before and after load; greedy output, tool calls (tool_choice: auto), image input (16–8,000 visual tokens per image, known-content OCR, counting and shapes) and video input (2 FPS sampling; known-content motion direction, colour order and frame counters) pass the Darkbloom qualification checks; teacher-forced perplexity on a 15,909-token panel is 7.314 under the serving policy.

Not measured: KL divergence vs BF16 and broad task benchmarks. These checks do not establish BF16 equivalence or quality on any workload.

License and attribution

• Original model and training: Z.ai, released under the MIT License. LICENSE is Z.ai's MIT license, copied byte-for-byte; keep it with any copy or substantial portion of these weights.
• NVFP4 quantization of the routed experts and dense MLPs: NVIDIA, with TensorRT Model Optimizer (ModelOpt 0.47, recipe nvfp4_experts_dense_mlp-kv_fp8_cast), published as nvidia/GLM-5.3-Flash-NVFP4 under the MIT license ("ready for commercial or non-commercial use"). NVIDIA states the model is not owned or developed by NVIDIA (third-party model).
• MLX packaging, non-expert 4-bit quantization and MTP-expert re-quantization: EigenLabs. This is not an official Z.ai or NVIDIA release.

artifact-provenance.json and SHA256SUMS.json record provenance and file hashes; nonexpert-quantization.json has per-tensor statistics for the affine matrices.