GLM-5.3-Flash EXL3 (~2.1 bpw, SM70 tile order)

PRs needed to run this on V100 (1Cat-vLLM): build main plus #1159 (decode fast paths), #1166 and #1165 (two more decode fast paths), #1161 (MTP startup fix), #1102 (EXL3 MoE loader and kernels) and #1125 (fast prefill; it already contains #1092, the FP32-softmax sparse-MLA kernel). #1091 (MTP speculative decoding) is already merged into main. Step-by-step build and serve commands are under Running it.

An EXL3 quantization of zai-org/GLM-5.3-Flash, made from the FP8 release. It is built to fit four 32 GB V100 GPUs and to run on the native SM70 EXL3 path in 1Cat-vLLM.

Parameters: 321B total, about 18B active per token, the same model as the base. The Hub's "Model size" figure (~52B) counts stored tensor elements, and an EXL3 tensor is stored as packed int16 trellis words that each hold 16 / K weights. Counted from the tensor shapes, the checkpoint holds 311.8B EXL3 expert weights and 9.6B unquantized ones.

Read this before downloading. The weights use a V100-friendly tile order, recorded as tile_order: sm70_colmajor in quantization_config. They run in 1Cat-vLLM on V100 with the PRs listed under Running it, which load this repo directly. Stock exllamav3 does not know this order and will decode the weights wrongly. exllamav3#475 adds the order to the converter, but this checkpoint predates its per-tensor marker, so even with #475 exllamav3 can't load it. A standard-order (sm80) build is not published.

Recipe

The overall bitrate excluding the head is 2.12 bpw. The codebook is mcg. Calibration used 352 rows of 8192 tokens. Mean per-layer relative reconstruction error over the MoE layers is 0.011 (worst layer 0.043). Expert Hessians blend the full-calibration Hessian with one built from the tokens actually routed to that expert, at weight 0.5, which cut held-out expert output error by about 14% in our tests.

Quality

Fidelity to the FP8 original. KL divergence and top-1 agreement of next-token predictions against the unquantized FP8 model, on the same three held-out texts (1,536 tokens each: a README, a design document and a Python file) for every build we made. Lower KL is better.

Task benchmarks. Same harness we used for our Qwen3.8-27B quants on V100: gsm8k (200 questions, strict match, 3,072-token answer budget), 75 factual questions, and 75 questions about things that do not exist (fabricated drugs, places, papers). Factual and fabrication answers are judged by grok-4.3. Greedy decoding.

It answered all 75 real questions correctly and declined 73 of the 75 fictional ones. GLM-5.3-Flash is a much larger model than a 27B, so the point here is that 2-bit quantization did not cost it that lead, not that it wins.

Agentic coding. On a small private set of three real bug-fix tasks graded by the projects' own tests, it passed two of three, the same as the cloud GLM-5.3-Flash API, at a lower judge score (17 vs 20 of 30 points). Earlier builds of ours collapsed into repetition loops at 75K-100K tokens of context on these tasks; this build, served with the attention fix below, did not in our runs (one run per task, so treat this as early evidence).

Speed

Four V100-SXM2-32GB (NVLink), TP4, one request at a time, FP8 KV cache. Measured on the kernels described below with a repacked copy of these weights; loading this repo directly through #1102 gives token-identical greedy output (8/8 prompts, no speculation).

Decode, mean of four 700-token chat responses, greedy:

The 86 -> 91 step (2026-10-10, second pass) comes from a profile of the 86 tok/s build: the KDA output norm ran as eight eager PyTorch kernels per layer because the custom-op dispatch takes the decomposed path and the SM70 decode graph is not compiled; the Triton gated RMSNorm is bit-identical and six times faster (#1165). The hyperconnection post-mapping and its logits dot product run as one native kernel for 1-8 tokens instead of two TileLang kernels with an FP32 intermediate (#1166). The 76 -> 86 step (2026-10-10) comes from three decode changes measured one at a time on the same build and prompts: the expert kernels read the trellis in a strip-major tile order with direct per-lane window loads and 8-deep prefetch (#1102, bitwise-identical output), the hyperconnection final stage runs natively for the 2-8 token verify batch, and the KDA input projection uses cuBLAS above one token (#1159). Greedy output on a 7-prompt set matches the previous kernels exactly on 4 prompts and diverges late (after 250+ characters) on the rest, the same pattern the previous build shows against its predecessor; 17x23 and the long-prompt summary are unchanged.

MTP numbers with 4-5 draft tokens are from an earlier build. Three draft tokens is the sweet spot, since each extra draft costs more step time than it adds in accepted tokens.

Prefill, time to first token, unique prompts (no prefix-cache reuse):

Running it on V100 (1Cat-vLLM)

Hardware: four V100-32GB (SXM2 with NVLink, or PCIe). TP4 needs all four. CUDA 12.8 toolkit, gcc/g++ 14.

The pieces

Build

git clone https://github.com/1CatAI/1Cat-vLLM && cd 1Cat-vLLM
git checkout ec60883ad                # tested base (2026-10-10 03:57 +0800); main from d4ce51399 on fails at MTP
                                      # draft-model init (Flash-V100 resources bound only at execution), see #1161
git fetch origin pull/1125/head:pr-1125 pull/1159/head:pr-1159 pull/1161/head:pr-1161 \
  pull/1166/head:pr-1166 pull/1165/head:pr-1165 pull/1102/head:pr-1102
# Cherry-pick, don't merge: #1125, #1159, #1161, #1166 and #1165 are rebased onto a newer main that contains
# d4ce51399, so merging their branches would pull that regression back in. Their own commits apply cleanly here.
git cherry-pick 587fac6eb da13d2d9c   # #1125 (contains #1092)
git cherry-pick 44d292046             # #1159 decode fast paths
git cherry-pick a66fcda49             # #1161 startup fix for MTP
git cherry-pick 74d2cf6da             # #1166 fused mHC post+dot
git cherry-pick 60dd820a7             # #1165 KDA output norm
git merge --no-edit pr-1102           # #1102 is based before d4ce51399; conflicts in CMakeLists.txt and csrc/ops.h
# Both conflicts are two PRs adding lines at the same spot; keep both sides:
sed -i '/^<<<<<<< \|^=======$\|^>>>>>>> /d' CMakeLists.txt csrc/ops.h
git add CMakeLists.txt csrc/ops.h && git commit --no-edit

uv venv --python 3.12 --seed
uv pip install -r requirements/build/cuda.txt --torch-backend=cu128
TORCH_CUDA_ARCH_LIST=7.0 CMAKE_CUDA_ARCHITECTURES=70 MAX_JOBS=10 \
CC=gcc-14 CXX=g++-14 CUDAHOSTCXX=g++-14 \
  uv pip install -e . --torch-backend=cu128 --no-build-isolation

The full build takes 40-60 minutes. The optional Rust vllm-server step needs protoc (package protobuf-compiler). Without it that step fails but the wheel still builds.

Check the build (the test deps are not part of the build requirements): uv pip install pytest tblib && .venv/bin/python -m pytest tests/kernels/quantization/test_exl3_moe.py tests/models/glm5next/test_sm70_sparse.py (80 pass).

Serve

Download the repo, then point --model at the local directory (that's the tested path):

hf download philbert440/GLM-5.3-Flash-EXL3 --local-dir ./GLM-5.3-Flash-EXL3
export CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3
export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0   # else 1.6 GiB is reserved for graph capture and no KV fits at 131K
.venv/bin/python -m vllm.entrypoints.openai.api_server \
  --model ./GLM-5.3-Flash-EXL3 --served-model-name glm-5.3-flash \
  --trust-remote-code --dtype half --tensor-parallel-size 4 \
  --kv-cache-dtype fp8_e4m3 --max-model-len 131072 --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 4096 --max-num-seqs 1 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"greedy"}' \
  --reasoning-parser deepseek_r1 --tool-call-parser glm47 --enable-auto-tool-choice

The startup log should show EXL3 routed experts: mcg, K=2 ... colmajor tile order (the loader reorders the tiles to the kernels' strip-major layout in place, nothing to configure) and GLM-5.3-Flash route: SM70 FP16 sparse MLA with packed E4M3FN KV and FP32-softmax decode/verify.

• Context: 131,072 tokens fits at --gpu-memory-utilization 0.92 with MTP, vision and FP8 KV, with the two settings above (--max-num-seqs, VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0); the KV pool is about 1.3 GiB per GPU and the weights take about 25.4 GiB.
• Concurrency: --max-num-seqs 4 for up to four users; it still fits at 131K and measured 143-156 tok/s aggregate (36-43 per user) with four 512-token greedy requests (measured before #1166/#1165), and 91 tok/s for a single user. Do not leave it unset: vLLM's default of 256 sequences makes the MLA prefill workspace one 2,304-token page per sequence (589,824 tokens, 9 GiB at TP4) and the server fails to start at 131K context.
• Prefix cache: it works in agentic multi-turn use (49-96% hit rates in our service), in 2,304-token blocks, a consequence of the hybrid KDA layers. Prompts shorter than one block are never cached.

Verified on our quad: loading this repo through #1102 gives greedy output identical to the repacked reference on 8/8 prompts (no speculation), and the 84K needle test passes. The PRs are drafts under review, so pin the commits you build. Known-good set (2026-10-10, the one the ~91 tok/s figure was measured on): main ec60883ad, #1125 at da13d2d9c (rebased, contains #1092 as 587fac6eb), #1159 at 44d292046, #1161 at a66fcda49, #1102 at b96d43e5c, #1166 at 74d2cf6da, #1165 at 60dd820a7. That stack, built as above and served with the command above, measured 91 tok/s decode (1 user, 512-token greedy responses) on our NVLink quad (re-verified 2026-10-10 from a clean clone with exactly these steps: 91.4-91.9 tok/s, 32.4 ms per pass); averaged over five varied 400-token prompts it does 85 tok/s at 32.7 ms per forward pass with 2.8 accepted tokens per pass. Under MTP the tok/s of any single prompt depends on its text (draft acceptance), so compare builds by milliseconds per pass, not by one prompt's tok/s; the same stack without the last two PRs measured 85.7 tok/s, 143-156 tok/s aggregate with four users, and retrieved a fact planted at the start of a 70K-token prompt (prefill 1,392 tok/s including generation).

Credits

Quantized with exllamav3 (MIT), using the SM70 work in Frenchy2k1's fork plus local patches. Base model by Z.ai, MIT licensed.