EXL3 quants of GLM-5.3-Flash
⚠️ Requires ExLlamaV3 v1.4.5 (or dev branch)
2.05 bits per weight
3.05 bits per weight
4.05 bits per weight
2.10 bits per weight
2.60 bits per weight
3.10 bits per weight
4.10 bits per weight
LocalMaxxing
vllm serve turboderp/GLM-5.3-Flash-exl3:3.05bpw --quantization exl3 --load-format exl3 -tp 2 -pp 2 --enable-expert-parallel --max-model-len 1048576 --max-num-seqs 8 --max-num-batched-tokens 1024 --gpu-memory-utilization 0.96 --limit-mm-per-prompt '{"image":0,"video":0}' -cc '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":5}' # Morrowmake image + Revamp import hookView run on LocalMaxxing vllm serve turboderp/GLM-5.3-Flash-exl3:3.05bpw --quantization exl3 --load-format exl3 -tp 2 --enable-expert-parallel --max-model-len 61440 --max-num-seqs 4 --max-num-batched-tokens 1024 --gpu-memory-utilization 0.97 --limit-mm-per-prompt '{"image":0,"video":0}' -cc '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":5}' # Morrowmake image + Revamp import hook, copy-in custom all-reduceView run on LocalMaxxing Median of 4 community runs on 2 setups, from LocalMaxxing.
Run it yourself: setup guide