GLM-5.3-Flash-Strata-Q2Q3-Experts

Experimental GGUF weights for running GLM-5.3-Flash on a normal PC with the LightTransport Strata fork: expert sidecars and a complete full model for a 128 GB host with a 16 GB NVIDIA GPU, a smaller REAP-224 model, and a complete REAP-50 model for a 64 GB host with a 16 GB AMD GPU. The sidecars requantize routed expert gate/up weights from IQ2_XS to Q2_K and down weights from IQ3_XXS to Q3_K; this changes weight values and is neither a lossless repack nor a fine-tune. REAP-224 keeps 224 of 288 routed experts per layer; REAP-50 keeps 144. Read the quality numbers below before choosing a model.

Current measurements, setup for one and two GPUs, and context-length limits are in docs/GLM53F_RELEASE.md on the fork's glm53f branch; bringing it up on other hardware is described in docs/GLM53F_NEW_HARDWARE.md. The conversion recipe is docs/GLM_RELEASE_Q2Q3.md.

Default since 2026-10-17: experts quantized from the FP8 release

On a 128 GB machine use GLM-5.3-Flash-Q23-FP8-assembled.gguf (or experts-q23-fp8.gguf with the Unsloth shards); on a 64 GB machine use reap50/GLM-5.3-Flash-REAP50-Q23-FP8-assembled.gguf. Their routed experts have the same types as Q23 (Q2_K gate/up, Q3_K down) but are quantized once, directly from the official FP8 weights, instead of from Unsloth's 2-bit experts (REAP-50: from its Q4_K_M experts). Measured against exact logits of the FP8 release on 16 held-out sequences (8,192 tokens):

Threadripper 1950X with an RX 9070 XT, the same decoder for every row: speed is 512 generated tokens after a 2K prompt; the full models use MTP depth 1 and affinity 0.08, the REAP-50 rows the B550 profile with unchanged routing inside a 60 GiB / no-swap cgroup; HumanEval as noted, REAP-50 with unchanged routing and MTP depth 1. The method, the determinism checks and the per-projection errors are in the fork's docs/GLM_FP8_REFERENCE.md. The first-release files below stay published for the earlier measurements.

Choosing for output quality

Use the full, unpruned GLM-5.3-Flash-Q23-assembled.gguf when output quality is the priority and the model fits in RAM. It retains all 288 routed experts per layer and is 120.9 GB. Q23 is still a quantized model. REAP-224 is a smaller, experimental 95.5 GB alternative that trades quality for memory headroom.

Full HumanEval results on the Threadripper 1950X, 128 GB DDR4-3200 and RTX 5060 Ti 16 GB used one greedy chat sample per task, low reasoning, a 2,048-token cap and isolated tests:

These runs use different builds and settings; their score differences do not isolate pruning or speculative depth. The fork's release floor is 152/164 plus a corpus gate (perplexity within +6%, KL within +0.08, top-1 within 3 points of the unpruned Q23 model). The REAP-224 tuned profile sits at that floor (152, then 150 on a rerun); the faster two-GPU configuration passes the corpus gate but not HumanEval.

For quality priority, keep routing unchanged and use strict MTP acceptance. Depth 2 has the highest recorded score above, but depth 1 was faster in a later matched 2K-prompt timing check: 20.28 versus 18.63 tok/s, with identical output on that prompt. The measured speed/quality compromise is full Q23 with affinity 0.08 and depth 1: about 23 tok/s, at the quality cost shown above. These timings are specific to the TR16 machine. See docs/GLM_Q2_DECODE_REDESIGN.md, docs/GLM_TR16_OPTIMIZATION.md and docs/GLM_RELEASE_20261013.md for the records.

Files

The sidecars are not a complete model. Keep all four original UD-Q2_K_XL shards from Unsloth; dense, attention, router, MTP and unconverted expert weights load from them. They need Strata's --expert-pack loader; other engines have not been tested.

Simplest option on a 128 GB machine: download GLM-5.3-Flash-Q23-FP8-assembled.gguf alone and pass it as the model (no --expert-pack).

Portability limitation of the sidecars. The loader fingerprints the original shards by size, header and file modification time, and there is no override, so a fresh download fails with source fingerprint mismatch. Either regenerate the sidecar against your shards (the fork's docs/GLM_Q2_REQUANTIZATION.md) or restore the original modification times (untested workaround, exact timestamps in docs/GLM_RELEASE_Q2Q3.md). The assembled REAP-50 file is self-contained and has no such check.

Measured results

Single-stream decode and prompt (prefill) rates, tokens per second. Full trials and conditions: docs/GLM53F_RELEASE.md in the fork.

Speeds are medians of timed 128-512-token continuations after a warm-up, with swap disabled, on the machines named; they are not promises for other hardware. Context length: about 26 MB of VRAM per 1K tokens on the primary GPU, taken from the GPU expert cache; roughly 128K is practical on a 16 GB card (see the fork's release notes).

Reproduce the conversions

Sidecars (128 GB machine, CUDA): calibrate the Unsloth model with the fork's corpus, then convert and verify.

git clone https://github.com/lighttransport/Strata.git && cd Strata
git checkout e5232ca3 && git submodule update --init --recursive
cmake -S . -B build-glm -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_MMQ_KQUANTS=ON
cmake --build build-glm -j 8 --target strata-glm-decode strata-glm-q2-pack strata-glm-assemble
hf download unsloth/GLM-5.3-Flash-GGUF --include 'UD-Q2_K_XL/*' --local-dir models/glm53f
MODEL=models/glm53f/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-00004.gguf; WORK=build-q2-redesign
python tools/glm_q2_corpus.py "$MODEL" "$WORK/corpus"
STRATA_GLM_CALIBRATION_DIR="$WORK/calibration" STRATA_GLM_LOCK_RUNTIME=1 \
  python tools/glm_q2_run_guard.py --output "$WORK/calibration-run" -- \
  numactl --interleave=all build-glm/strata-glm-decode "$MODEL" 0 1 4096 15 \
  --context=8192 --prefill-batch=256 --gpu-budget-mib=12288 --cpu-affinity=numa --decode-graphs \
  --eval-corpus="$WORK/corpus/calibration.ids"
build-glm/strata-glm-q2-pack "$MODEL" experts-q23.gguf --calibration="$WORK/calibration" --threads=16
build-glm/strata-glm-q2-pack "$MODEL" experts-q22.gguf --calibration="$WORK/calibration" --threads=16 --down=q2 --reuse=experts-q23.gguf
build-glm/strata-glm-q2-pack --verify "$MODEL" experts-q23.gguf

FP8-derived files (the defaults): download zai-org/GLM-5.3-Flash and add --fp8=<its directory> --gate-type=q2_k --down-type=q3_k to the strata-glm-q2-pack command above, then strata-glm-assemble "$MODEL" experts-q23-fp8.gguf GLM-5.3-Flash-Q23-FP8-assembled.gguf. For REAP-50, recover the kept experts with tools/glm_reap_expert_map.py (every REAP-50 router row equals exactly one original row), pack against the published REAP-50 file with --expert-map=, and assemble with strata-glm-assemble --reassemble. Full commands: docs/GLM_RELEASE_Q2Q3.md, "Reproducing the FP8-derived weights".

REAP-50 assembled model (first release): the same calibration and strata-glm-q2-pack run on GLM-5.3-Flash-REAP50-Q4_K_M.gguf from patrickbdevaney/GLM-5.3-Flash-REAP50-GGUF, followed by strata-glm-assemble REAP50-Q4_K_M.gguf reap50-q23.gguf GLM-5.3-Flash-REAP50-Q23-assembled.gguf, which copies bytes without requantizing and verifies every expert checksum.

These commands reproduce the workflow, not the published bytes; identical output needs the same revision, corpus, settings and toolchain.

Run

128 GB machine: use an assembled file as the model. The default weights were measured with configs/experimental/glm53f-q23fp8-tr16-9070xt-aff08.json (and -aff0.json for unchanged routing). Other profiles in the fork: configs/experimental/glm53f-q23-reap224-tr16-tuned.json (REAP-224, one GPU), configs/experimental/glm53f-reap224-tr16-two-gpu-decode.json and …-prefill.json (two GPUs, run through tools/glm_mixed_run.py), and configs/experimental/glm53f-q23-tr16-aff08.json (full model; with the sidecars instead, pass the first Unsloth shard and --expert-pack=/path/to/experts-q23.gguf). Profiles work with python3 -m serve.server --engine glm --config …. 64 GB machine: use reap50/GLM-5.3-Flash-REAP50-Q23-FP8-assembled.gguf (the default; or the first REAP-50 file) as the model with configs/glm53f-reap50-q23-b550-60g-9070xt-experimental.json via tools/glm_b550_60g.sh start, inside the 60 GiB memory cgroup it sets up; with an 8 GB card, the configs/glm53f-reap50-q23-64g-rtx3070-*.json presets. Full command lines are in docs/GLM53F_RELEASE.md and docs/GLM_RELEASE_Q2Q3.md.

License and attribution

The original GLM-5.3-Flash is by Z.ai under the MIT license, copyright (c) 2026 Z.AI Co., Ltd. The UD-Q2_K_XL quantization is by Unsloth (MIT). The REAP-50 pruned GGUF base is by patrickbdevaney (MIT, base model zai-org/GLM-5.3-Flash). The original Strata engine is by Niko1221 and the Strata contributors (MIT); this conversion uses the LightTransport fork. Quantization kernels come from GGML (MIT). These derived weights are distributed under MIT; retain the upstream notices when redistributing. This is a community conversion, not an official Z.ai, Unsloth, REAP or upstream Strata release.