GLM-5.3-Flash RANMA EXL3 GGUF

[!WARNING] Requires ranma.cpp ranma_20261007 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

[!NOTE] No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 quantizations made by turboderp of zai-org/GLM-5.3-Flash into GGUF files that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. The fused KDA qkv_proj (and its conv1d) of the checkpoint is stored as the separate q, k and v tensors that the GGUF graph reads, cut at 128-row boundaries without changing the trellis bytes. All credit for the quantization goes to turboderp and the ExLlamaV3 project. Thank you.

Sources (EXL3, by turboderp), one branch per quantization. Each source branch maps to one folder in this repository:

Both were produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5/h6 are the head bits (output layer).

The multi-token-prediction (MTP) layer of the checkpoint is included in the same file; see Running.

Not included

In llama.cpp the vision encoder is a separate mmproj file, so it can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors (llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically. manifest.json lists the size and SHA-256 of every file.

File names: GLM-5.3-Flash-EXL3-<bpw>-<head>-0000K-of-0000N.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the example below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only; there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement). The first shard is enough for -m; the other shards are found by name.

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights (about 5.5 GiB in VRAM for 3.05 bpw), the KV cache and the compute buffers. Most layers of this model are linear-attention (KDA) layers with a fixed-size state, so the KV cache is small: about 1.2 GiB at -c 65536 (f16). With L1 = 20480 at -c 65536 the dedicated VRAM peaked at about 28.6 GiB on a 32 GB R9700. Keep a margin of a few GiB, more if the GPU also drives a display. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.
• L2. The routed experts take 71.3 GiB (2.05 bpw) and 106.7 GiB (3.05 bpw). In exclusive mode L1 and L2 hold different experts, so L1 + L2 at or above that size reads nothing from the SSD. The physical RAM in use is about L2 plus a few GiB; Windows reports a higher private memory, because it also counts the VRAM and the mapped host memory. The server refuses to load if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.

The command and values below are an example only. Pick them for your own use and system: the context length, the split between L1 and L2, whether to use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example command (64K context)

32 GB VRAM + 128 GB RAM:

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h6\GLM-5.3-Flash-EXL3-3.05bpw-h6-00001-of-00004.gguf ^
  -c 65536 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 20480 --expert-l2-mib 92160 ^
  --expert-profile-dir .\expert-profiles

For 2.05 bpw use EXL3-2.05bpw-h5\GLM-5.3-Flash-EXL3-2.05bpw-h5-00001-of-00003.gguf; its experts fit in L1 + L2 entirely, so a smaller L2 is enough. For MTP add --spec-type draft-mtp --spec-smart-store .\smart-store.json and lower L1 to --expert-l1-mib 18944.

The chat template of the original model is in the file. Its prompts start with [gMASK]<sop>; use the chat endpoint (or add that prefix yourself for raw completion).

Details of every option: expert-cache.md, expert-cache-exclusive.md, expert-cache-l2.md, expert-cache-joint.md, host-direct-moe.md, spec-smart.md, gpu-heartbeat.md, exl3.md.

License

Same as the original model: MIT, see the original repository.