GLM-5.3-Flash-NVFP4

Model Overview

• Model Architecture: Glm5NextForConditionalGeneration
  • Input: Text / Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP4
  • Activation quantization: FP4
• Release Date: 2026-08-27
• Version: 1.0
• Model Developers: RedHatAI

This model is a quantized version of zai-org/GLM-5.3-Flash. It was evaluated on several tasks to assess its quality.

Model Optimizations

This model was obtained by quantizing the MoE expert weights and activations of zai-org/GLM-5.3-Flash to FP4 (NVFP4) data type, ready for inference with vLLM. The MTP layers are kept in FP8, matching the source checkpoint.

This optimization reduces the number of bits per parameter in the quantized layers from 8 to 4, reducing the disk size and GPU memory requirements by approximately 50% of the quantized weights.

Only the weights and activations of the MoE expert linear operators are quantized using LLM Compressor.

Deployment

vLLM Serving

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  vllm/vllm-openai:glm53-flash RedHatAI/GLM-5.3-Flash-NVFP4 \
  --tensor-parallel-size 4 \
  --no-enable-flashinfer-autotune \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45

Enable Speculative Decoding

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  vllm/vllm-openai:glm53-flash RedHatAI/GLM-5.3-Flash-NVFP4 \
  --tensor-parallel-size 4 \
  --no-enable-flashinfer-autotune \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --gpu-memory-utilization 0.85 \
  --disable-custom-all-reduce \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Creation

This model was created by applying LLM Compressor with the NVFP4 scheme, exported in compressed-tensors format.

Evaluation

This model was evaluated on GSM8K Platinum, MATH-500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, all served with vLLM (OpenAI-compatible API). Each benchmark was run with 3 seeds (8 seeds for AIME 2025) and the results averaged.

Accuracy